Taghreed: A System for Querying, Analyzing, andVisualizing Geotagged Microblogs∗

Amr Magdy#§, Louai Alarabi#§, Saif Al-Harthi§, Mashaal Musleh§,Thanaa M. Ghanem⋆§, Sohaib Ghani§, Mohamed F. Mokbel#§

§KACST GIS Technology Innovation Center, Umm Al-Qura University, Makkah, KSA#Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN

⋆Department of Information and Computer Sciences, Metropolitan State University, Saint Paul, MN{amr,louai,mokbel}@cs.umn.edu, {sharthi,mmusleh,sghani}@gistic.org,

thanaa.ghanem@metrostate.edu

ABSTRACT

This paper presents Taghreed; a full-fledged system for effi-cient and scalable querying, analyzing, and visualizing geo-tagged microblogs, e.g., tweets. Taghreed supports arbitraryqueries on a large number (Billions) of microblogs that goup to several months in the past. Taghreed consists of fourmain components: (1) Indexer, (2) query engine, (3) re-covery manager, and (4) visualizer. Taghreed indexer effi-ciently digests incoming microblogs with high arrival ratesin light memory-resident indexes. When the memory be-comes full, a flushing policy manager transfers the mem-ory contents to disk indexes which are managing Billions ofmicroblogs for several months. On memory failure, the re-covery manager restores the system status from replicatedcopies for the main-memory content. Taghreed query en-gine consists of two modules: a query optimizer and a queryprocessor. The query optimizer generates an optimal queryplan to be executed by the query processor through efficientretrieval techniques to provide low query response, i.e., orderof milli-seconds. Taghreed visualizer allows end users to issuea wide variety of spatio-temporal queries. Then, it graphi-cally presents the answers and allows interactive explorationthrough them. Taghreed is the first system that addressesall these challenges collectively for microblogs data. In thepaper, each system component is described in detail.

Categories and Subject Descriptors

H.2.8 [Database Applications]: Spatial databases andGIS

Keywords

Microblogs, Spatio-temporal, Indexing, Query Processing

∗This work is supported by KACST GIS Technology In-novation Center at Umm Al-Qura University, under projectGISTIC-13-06, and was done while the first, second, fifth,and last authors were visiting the center.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Requestpermissions from Permissions@acm.org.SIGSPATIAL ’14, November 04-07 2014, Dallas/Fort Worth, TX, USACopyright 2014 ACM 978-1-4503-3131-9/14/11$15.00http://dx.doi.org/10.1145/2666310.2666397

1. INTRODUCTIONOnline social media services become very popular in the

last decade which has led to explosive growth in size ofmicroblogs data, e.g., tweets, Facebook comments, andFoursquare check in’s. Everyday, 255+ Million active Twit-ter users generate 500+ Million tweets [40, 42], while 1.23+Billion Facebook users post 3.2+ Billion comments [13].As user-generated data, microblogs form a stream of richdata that carries different types of information includingtext, location information, and users information. More-over, microblogs textual content is rich with user updates onreal-time events, interesting keywords/hashtags, news items,opinions/reviews, hyperlinks, images, and videos. Conse-quently, this richness in data enables new queries and ap-plications on microblogs that were not applicable earlier ontraditional data streams. For example, performing keywordsearch on streaming data [7, 9, 45, 46] is now of interest forthe first time. Other examples for newly emerging applica-tions on microblogs includes event detection [1, 20, 27, 36,44], news extraction [6, 34, 37], spatial search [24], and anal-ysis [17, 38, 39]. Such kinds of applications are so importantthat major IT companies spend millions of dollars to enablethem to their customers [3, 41].

Although the research community has addressed a largeset of newly emerging queries and applications on mi-croblogs [1, 7, 12, 21, 24, 26, 27, 34, 37, 45], none of thoseprovides a full fledged system that facilitates microblogsdata management to support arbitrary queries on multipleattributes. Only TweeQL [26, 27] is proposed as a gen-eral query language for Twitter data, a prime example ofmicroblogs. However, TweeQL just provides a wrapping in-terface for Twitter streaming APIs without addressing theactual data management issues for microblogs big data. Onthe other hand, microblogs can be considered a kind of semi-structured data that could be managed with systems like As-terixDB [2]; a distributed system that supports interactivequeries on big semi-structured data. However, AsterixDB isnot appropriate to handle microblogs in its streaming formfor two main reasons: (a) AsterixDB does not support di-gesting and indexing fast streaming data in real-time, andconsequently (b) AsterixDB does not provide mechanismsto manage data flushing for memory-resident data to diskwhich is a must for managing a long history of microblogsdata. Moreover, none of this work addresses interactive vi-sualization and analysis with end users.

163

In this paper, we present Taghreed; a full-fledged systemfor efficient and scalable querying, analyzing, and visualiz-ing geotagged microblogs. On the contrary to all other workon microblogs, Taghreed system goals is to manage, query,analyze, and visualize microblogs streaming data for longperiods that go up to several months. In addition, Taghreedprovides data management techniques that are able to sup-port interactive query responses on different microblogs at-tributes, promoting spatial, temporal, and keyword as first-class attributes as they are involved in most of the impor-tant queries and application of microblogs. The main tech-nical challenges in Taghreed comes from three main sources:(1) the large number of microblogs to be managed (orderof Billions), (2) the continuous arrival of new microblogs inhigh rates (order of hundreds or thousands every second),and (3) the new types of queries on the rich microblogs datawhich cannot be supported by Data Stream ManagementSystems (DSMS). Unlike the traditional streaming data, thecombination of fast-arriving large number of data items andthe queries on a rich set of attributes, e.g., spatial, tem-poral, keyword, users, and language, requires Taghreed toemploy both real-time and disk-based efficient and scalabledata indexing to be able to digest real-time data and supportsuch queries interactively, i.e., the query answer is providedinstantly. Thus, Taghreed main components are: (1) Ef-ficient and scalable data indexing, (2) efficient interactivequery processing, (3) low-overhead recovery management,and (4) effective data visualization, where each of them facesdifferent technical challenges.Indexer. Supporting indexing on microblogs data faces

three main challenges: (1) continuous digestion of real-time microblogs, with high arrival rates, in main-memory,(2) managing a large number of microblogs, order of Billions,on disk, and (3) managing data flushing from main-memoryindexes to disk indexes so that the system response is nothurt and the main-memory resources are efficiently utilized.To address these challenges, Taghreed proposes a set of seg-mented hierarchical indexes on spatial, temporal, and key-word attributes of microblogs. In both main-memory anddisk, Taghreed employs two indexes: a keyword index and aspatial index. Each index consists of temporally-partitionedindex segments, where each segment organizes its data basedon either spatial or keyword attributes, depending on theindex type. Based on the different challenges of indexingreal-time data and big historical data, the organization andthe employed data structure of main-memory and disk in-dexes are different. Taghreed also adapts several flushingpolicies to transfer data from main-memory to disk. Theflushing manager is mainly concerned with minimizing thesystem overhead, e.g., disk access latency and main-memoryutilization, while keeping as much data as possible in main-memory to reduce the query processing time. The details ofTaghreed indexing are presented in Section 4.Query engine. Taghreed query engine is mainly con-

cerned with supporting a wide set of generic interactivequeries, i.e., low query response in order of milli-seconds,on the large number of managed microblogs. By genericqueries we mean a framework that is not tailored to certainpredefined queries. Instead, it could be easily adapted tosupport different types of queries with almost no loss in sys-tem performance. To this end, Taghreed has chosen to sup-port efficient retrieval of individual microblogs based on themain querying attributes which are the spatial, temporal,

and keyword attributes. In other words, Taghreed can geta set of microblogs that lie within certain spatio-temporalrange and satisfy certain keyword expression very fast. Us-ing these individual microblogs, other types of filtering, e.g.,based on users, and aggregation, e.g., frequent keywords,are performed with efficient distributed data scanners. Toaccomplish low query responses, the query engine consistsof two main modules: (1) A query optimizer, whose maintask is to generate an optimal query plan, to hit the sys-tem indexes, based on different cost models. (2) A queryprocessor, that performs effective pruning through the sys-tem indexes to efficiently retrieve the data of interest withminimal system overhead. Then, it employs efficient dis-tributed data scanners to answer more complex or extendedqueries that involve attributes other than spatial, tempo-ral, and keywords. The details of Taghreed query engine arepresented in Section 5.

Recovery manager. With hours and even days of datamanaged in the main-memory, Taghreed provides a recoverymanagement module that is able to restore the system sta-tus on memory failures. For social media application, therecovery manager is required to be of low-overhead so thatit does not hurt the continuous real-time operations. Thus,Taghreed employs a memory-based triple-redundancy modelthat replicates the main-memory contents three time. Onmemory failure, the recovery manager uses the replicatedcopies to restore the system status without interrupting thereal-time operations. The details of Taghreed recovery man-ager are presented in Section 6.

Visualizer. With all the technical details of manag-ing and querying microblogs data in the system back end,Taghreed provides end-to-end solution with an interactivefront end that takes users queries through web-based inter-faces, dispatches them to the query engine, and receive theanswers back to visualize. The visualization module com-prises of an integrated interface that present answers of arich set of queries and also provides a comprehensive appli-cations on microblogs data. The details of Taghreed visual-izer are presented in Section 7.

2. RELATED WORKTaghreed related work lies mostly in three areas, namely,

microblogs data processing, big data systems, and spatial-keyword search.

Microblogs data processing. Microblogs researchmostly lies in two categories: (a) Microblogs data anal-ysis where the main focus is to provide novel applicationsand exploit user activities and opinions from microblogs con-tents, e.g., semantic and sentiment analysis [5, 29, 31], de-cision making [8], news extraction [37], event and trend de-tection [1, 20, 27, 36], understanding the characteristics ofmicroblog posts and search queries [21, 35], microblogs rank-ing [12, 43], and recommending users to follow or news toread [15, 34]. (b) Microblogs data management wherethe main focus is to provide infrastructure to handle mi-croblogs data, e.g., logging [19], indexing [7, 9, 24, 45, 46],and query languages [26]. Taghreed lies in the second cat-egory. Distinguishing itself from all existing data manage-ment work on microblogs, Taghreed is the first system to:(1) Combine keyword search with spatio-temporal search onmicroblogs providing all the needed indexing, query opti-mization, and processing infrastructure techniques. (2) Pro-vide flexible querying framework on other microblogs at-

164

tribute, e.g., users and language. (3) Handle microblogsdata in both real-time and historical forms so that queriesare enabled on a long history of data that goes up to severalmonths.Big data systems. Microblogs data is a kind of big data

that is flowing in a streaming form in large numbers andwith high arrival rates. Although there exists a lot of workon spatio-temporal queries over streaming data [18, 22, 30,32, 47], the main focus of such work is on continuous queriesover moving objects. In such case, a query is registered first,then its answer is composed over time from the incomingdata stream. On the contrary, queries answers on microblogsare retrieved from existing stored objects that have arrivedprior to issuing the query. On another hand, AsterixDB [2]and Map-D [25] are systems that support interactive querieson big data. However, none of them tackle the problem ofreal-time digestion and indexing of data with high arrivalrates. They are mostly designed for managing big batchesof stored data. Consequently, they lack the ability to handlethe microblogs in its mixed form where the data comes asa fast stream, managed as hot recent data for a while, andthen being flushed to a large data repository on disk.Spatial-keyword search. Spatial keyword queries of

different types are well studied on web documents and webspatial objects (see the survey [10]). However, all these tech-niques are not suitable for handling microblogs for two rea-sons: (1) none of these techniques consider the temporaldimension which is a must in all microblogs queries, and(2) these techniques mostly use offline disk-based data parti-tioning indexing which cannot scale to support the dynamicnature and arrival rates of microblogs [7, 10].

3. SYSTEM OVERVIEWThis section gives an overview of Taghreed design princi-

ples, system architecture, and supported queries.

3.1 Design PrinciplesTaghreed is designed based on two principles:1. Dominance of the temporal, spatial, and key-

word attributes: All microblogs queries have to betemporal, and then it mostly involves spatial and key-word dimensions. The real-time nature of microblogsdata makes the temporal dimension a must to be in-cluded in all queries (see [7]). In addition, the richestattributes in microblogs are the spatial and keywordsattributes which are involved in most of the queries andapplications (see [1, 7, 12, 21, 24, 26, 27, 34, 37, 45]).Consequently, Taghreed promotes the three attributes,spatial, temporal, and keyword, as first-class citizensand supports native system indexes on them. In ad-dition to their importance, indexing spatial, temporal,and keyword attributes provides effective pruning forthe microblogs search space.

2. Importance of queries on recent microblogs:Due to the rich real-time content of microblogs, e.g.,real-time updates on ongoing events [6, 11], impor-tant queries are posted on recent microblogs data,i.e., data of the last few seconds, minutes, or hours.This triggers all the work on handling queries on real-time microblogs ( see [1, 7, 23, 24, 28, 31, 34, 36,45]). Consequently, Taghreed is designed to supportqueries on real-time microblogs with all its subsequentrequirements of main-memory indexing, recovery man-agement, and flushing management.

Interactive Visualizer

Query

Visualizer

User

Query

Optimizer

Query

Processor

Query Plan

Query Dispatching

Query Engine

Query Answer

Answer

Disk Indexer

Main-memory

Indexer

Flushing Policy

Microblogs Stream

Indexer

Main-memory

Contents

Recovery Module

Recovery

Manager

PreprocessorPreprocessed Microblogs

Memory Failure

Figure 1: Taghreed Architecture.

3.2 System ArchitectureFigure 1 gives Taghreed system architecture that consists

of four main components, namely, indexer, query engine,recovery manager, and visualizer. We briefly introduce eachof them below.

Indexer. Taghreed indexer is the main component thatis responsible for handling the microblogs data. First, thereal-time microblogs are going through a preprocessor thatperforms location and keyword extraction. Then, the real-time microblogs are continuously digested in main-memoryindexes, on spatial, temporal, and keywords attributes, thatprovide high digestion rates and effective pruning for thesearch space. When the memory becomes full, a subset ofthe main-memory microblogs are selected, through a flush-ing policy module, to be consolidated into scalable disk in-dexes that are able to manage a large number of microblogsfor long periods, go up to several months. The details ofTaghreed indexer are presented in Section 4.

Query Engine. Taghreed query engine consists of twomain sub-components: a query optimizer and a query pro-cessor. The query optimizer takes a user query from thesystem front end dispatcher, generates an optimized queryplan to hit the system indexes efficiently, and provides thequery plan to the query processor to execute. The queryprocessor takes the query plan, executes it, and feed thequery answer to the system front end to be visualize to endusers. The details of Taghreed query engine are presented inSection 5.

Recovery Manager. With dense main-memory con-tents, Taghreed accounts for memory failures by incorpo-rating a recovery manager component. The recovery man-ager employs a triple-redundancy model for backing up themain-memory contents. When the memory fails, the backupcopies are used to restore the system status. The details ofTaghreed recovery manager are presented in Section 6.

Visualizer. As a full-fledged system that provides end-to-end solution for querying microblogs, Taghreed providesan interactive visualizer component that interacts with endusers. Taghreed is among few data management systemsthat take into account the end user perspective in its design.Such design strategy is very important for designing socialmedia applications as the user experience plays a vital rolein application usability and popularity [16]. The visualizerallows the user to issue queries, dispatches them to the queryengine, and receive the answers to present them to end usersin graphical form. It provides Taghreed users with a rich setof interactive queries which are presented in an integratedinterface. The details of Taghreed visualizer are presented inSection 7.

165

3.3 Supported QueriesTaghreed supports any query on microblogs that involves

the spatial, temporal, and keyword attributes. The tem-poral dimension is mandatory while spatial and keyworddimensions are optional. The queries are answered by fil-tering the search space through hitting the system indexesfor the three main attributes. If the query involves otherattributes, Taghreed employs generic distributed data scan-ners that refine the answer based on the other attributes.This enables Taghreed to support generic queries that sat-isfy a wide variety of applications. Section 7 presents a richsample of queries that can be supported by Taghreed systemalong with their interactive visual interfaces to end users.

4. INDEXERThe plethora of microblogs data combined with the newly

motivated queries on the rich stream, e.g., spatial-keywordsearch, makes data indexing an essential part to manage andquery microblogs data. In other words, microblogs come invery large numbers, i.e., Millions and even Billions, so thatinteractive queries—where the answer is needed instantly—cannot be answered efficiently by just scanning the big data.Thus, supporting indexing on microblogs is essential for ef-ficient retrieval of microblogs data that is needed to answerthe interactive queries. Next in this section, we will dis-cuss the selection of attributes to be indexed along with theorganization and the details of the indexes.As a user-generated data, microblogs are rich data that

contains several attributes to post queries on, e.g., times-tamp, location, keywords, user attributes, and language. Al-though Taghreed goal is to provide a framework to answerarbitrary queries on microblogs where more queries can besupported with minimal performance loss, it is not practicalfrom a system point of view to support a separate index oneach of microblogs attribute. Thus, Taghreed has chosen toprovide indexes on the most effective attributes to be in-dexed and perform efficient distributed scanning for otherattributes if involved in any of the queries. The effectiveattributes to be indexed are chosen based on two factors:(1) the attributes that are involved in most of the impor-tant queries on microblogs, (2) the attributes whose a widedomain of values, i.e., can take a large number of distinctvalues, and hence would provide the most effective pruningfor the microblogs search space. An example of an attributethat has limited domain of values is the language attribute.The language attribute in Twitter data has a domain of only56 values. If language attribute is indexed, that means thatall of the incoming microblogs, that are Billions, will be di-vided into at most of fifty six chunks where each of themwill be huge to search within. Thus, the language attributewill not provide very effective pruning for Billions of mi-croblogs. Based on these two factors, Taghreed has chosento support indexes for spatial, temporal, and keywords at-tributes. From one hand, most of microblogs queries andapplications in the literature involve these three attributes.On the other hand, the domains of values of these attributesare wide enough to provide effective pruning for the searchspace so that the number of processed microblogs to answercertain query is minimized.As a fast stream that comes with rapidly increasing high

arrival rates, microblogs real-time digestion must be in lightmain-memory indexes that is able to cope with the in-

Segment m

(m-1)T-mT hr ago data

Segment 3

2T-3T hr ago data

Segment 2

T-2T hr ago data

Segment 1

0-T hr ago dataPreprocessed

Microblogs

......

Segment m

(m-1)T-mT hr ago data

Segment 3

2T-3T hr ago data

Segment 2

T-2T hr ago data

Segment 1

0-T hr ago data

......

...

...

...

...

Keyword Index

Spatial Index

Figure 2: Taghreed In-memory Segmented Index.

creasing arrival rates (Twitter rate in October 2013 was5,200 tweet/second compared to 4,000 tweet/second in April2012 [42]). In addition, keeping the most recent data inmain-memory speeds up the query responses as most of thereal-world queries access the most recent data [7]. On theother hand, Taghreed would not be able to manage Billions ofmicroblogs, for several months, in main-memory due to thescarcity of this resource. Consequently, Taghreed indexingwould include both main-memory and disk-resident indexes.As Figure 1 shows, the streaming microblogs are handledby a real-time preprocessor for keyword and location ex-traction. Then, the main-memory indexes would digest thepreprocessed microblogs in real-time with high arrival rates.When the memory becomes full, a flushing manager wouldmanage to transfer main-memory contents to the disk in-dexes. Disk indexes are then responsible to manage a verylarge number of microblogs for several months.

In the rest of this section we describe Taghreed indexer.First, we describe the main-memory indexing of real-timemicroblogs in Section 4.1. Second, we describe the disk-based indexing in Section 4.2. Finally, the flushing manage-ment from main-memory to disk is discussed in Section 4.3.

4.1 Main-memory IndexingTaghreed provides light-weight in-memory indexes that

employ efficient index update techniques to be able to digesthigh arrival rates of microblogs in real-time. The indexes aresupported on the spatial, temporal, and keywords attributes.In this section, we describe index structure and organization.We also discuss the real-time digestion through efficient in-dex update operations.

Index organization. Figure 2 shows the organizationof Taghreed in-memory indexes. Taghreed employs two seg-mented indexes in the main-memory: a keyword index and aspatial index. Both of them are temporally partitioned intosuccessive disjoint index segments. Each segment indexesthe data of T hours, where T is a parameter to be opti-mized by the flushing manager (see Section 4.3). The newlyincoming microblogs are digested in the most recent segment(Segment 1 in Figure 2). Once the segment spans T hoursof data, the segment is concluded and a new empty segmentis introduced to digest the new data. Index segmentationhas two main merits: (a) new microblogs are digested in asmaller index, which is the most recent segment, and hencebecomes more efficient, and (b) it eases flushing data from

166

memory to disk under certain flushing policies.Real-time digestion. The keyword index segment is an

inverted index that organizes the data in a light hashtable.The hastable maps a single keyword (the key) to a list ofmicroblogs that contain the keyword. The list of microblogsof each keyword is ordered reverse-chronologically so thatthe insertion is always in the list front item in O(1). Thus,the whole insertion in the index is O(1). With such opti-mization, Taghreed keyword segments are able to digest upto 32,000 microblog/second.The spatial index segment is a pyramid index structure [4]

(similar to a partial quad tree [14]) that employs efficientupdate and structuring techniques to provide a light-weightspatial indexing. The pyramid index is a space-partitioningtree of cells, where each cell has either zero or four chil-dren cells, and sibling cells cover equal spatial areas. Unlikedata-partitioning indexes, e.g., R-tree, pyramid index couldsupport high digestion rates due to the low restructuringoverhead with newly incoming data. Each cell has a ca-pacity of certain number of microblogs, where the capacityis a system parameter. The microblogs inside each cell arestored in a reversed chronological order. When a cell en-counters microblogs that exceeds its capacity, it is split intofour children cells if and only if the microblogs lie in at leasttwo different quarters of the cell. This excludes redundantsplit operations for highly skewed data. Similarly, to elimi-nate redundant merge operations, underutilized cells are notmerged immediately. Instead, four siblings are merged, inlazy basis, only when three of them are completely empty.Such lazy split and merge operations saves 90% of the struc-turing operations in the highly dynamic microblogs envi-ronment. Also the index structure stabilizes relatively fastwhich decrease the structuring overhead to its minimal lev-els. To update the index efficiently, microblogs are insertedin batches, periodically each t seconds, instead of travers-ing the pyramid levels for each individual microblog. Typi-cal values of t are 1-2 seconds so that several thousands ofmicroblogs are inserted in each batch. Then, the pyramidlevels are traversed once with minimum bounding rectangleof all the microblogs in the batch. This saves thousands ofcomparison operation in each insertion cycle. With these op-timization, Taghreed spatial segments are able to digest upto 32,000 microblog/second. More details on the real-timespatial indexing in Taghreed can be reviewed in [24].All the deletion operation from the in-memory indexes are

handled by Taghreed flushing manager. Thus, the detailsof deletion from the main-memory indexes are explained inSection 4.3.

4.2 Disk-based IndexingTo be able to support a large number of microblogs data

for long periods, that go up to several months, Taghreedsupports disk indexes to manage the microblogs that are ex-pelled from the main-memory. Similar to the main-memoryindexes, disk indexes are supported on spatial, temporal,and keywords attributes. However, the disk indexes orga-nization and structure are different from the main-memoryones. In this section, we describe the index organization,structure, and update operations.Index organization. Taghreed employs two disk indexes:

a keyword index and a spatial index. Figure 3 shows theorganization of Taghreed disk spatial index, which is simi-lar to the organization of the keyword index as well. Each

Flushed

Microblogs

Segment n

1 month data

Segment 1

1 month data

......Monthly Segments

Daily Segments

Segment 1

1 day data

Segment 2

1 day data

Segment 3

1 day data

Segment 4

1 day data

Segment 5

1 day data

Segment 6

1 day data

Segment 7

1 day data

Weekly Segments

Segment 3

1 week data

Segment 2

1 week data

Segment 1

1 week data

Segment 4

1 week data

Figure 3: Taghreed Disk Spatial Index.

of both indexes is organized in temporally partitioned seg-ments. The temporal segments are replicated in a hierarchyof three levels, namely, daily segments, weekly segments,and monthly segments. The daily segments level stores thedata of each calendar day in a separate segment. The weeklysegments level consolidates the data in each successive sevendaily segments that forms data for one calendar week in asingle weekly segment. Again, the monthly segments levelconsolidates data of each successive four weekly segments ina single segment that manages the data of a whole calen-dar month. The main reason behind replicating the indexeddata on three temporal levels is to minimize the number ofaccessed index segments while processing queries for differ-ent temporal periods. For example, for an incoming queryasking about data of two months, if only daily segmentsare stored, then the query processor would need to accesssixty indexes to answer the query. On the contrary to thedescribed setting, the query processor would need to ac-cess only two indexes of the two months time horizon of thequery. This significantly reduces the query processing timeso that Taghreed is able to support queries on relatively longperiods.

Index structure and update. Taghreed disk keywordindex segments structure are inverted indexes while spatialindex segments are R+-trees. Unlike pyramid structure, R+-tree is disk-friendly where tree nodes are disk pages. At anypoint of time, only a single daily segment is active to absorbthe expelled microblogs from the main-memory. Once a onefull day passes, the current active daily segment is concludedand a new empty segment is introduced to absorb the nextincoming data. Upon concluding seven successive daily seg-ments, a weekly segment is created in the background tomerged the data of the whole week in a single index. Thesame repeats for monthly segments upon creating four suc-cessive weekly segments.

Updating the disk indexes is performed in isolation fromthe employed flushing policy (see Section 4.3 for flushing

167

policies details). Regardless the flushing policy, disk indexesare always temporally disjoint from the main-memory in-dexes. In other words, whenever Taghreed inserts data inthe disk indexes, there exists a check point timestamp tcpwhere all the data in the main-memory indexes are morerecent than tcp and all the data in disk-based indexes areolder than or equals tcp. Guaranteeing this, consolidatingdata from main-memory keyword index into disk keywordindex is done very efficient through bucket-to-bucket map-ping without bothering with deforming the temporal organi-zation of the data. It is worth noting that mapping memorybuckets to disk buckets is inapplicable for the spatial indexas the spatial structure that is employed in the main-memory(the pyramid index) is space-partitioning structure which isdifferent than the data-partitioning structure (the R+ tree)that is employed on disk. In the rest of the section, we ex-plain the consolidation process from main-memory to diskfor both indexes, keyword and spatial.To consolidate data from main-memory keyword index to

disk keyword index, we perform three steps. First, we checkif the new data would require creating a new daily segment.For this, we check the oldest and the newest timestamps ofthe data to be flushed, which are provided from the flushingmanager. If the two timestamps span two different days,then a new daily segment is created. The second and thirdsteps are performed repeatedly for each keyword slot in theindex. Each slot contains a list of microblogs that are storedin a reverse chronological order. The second step maps eachslot from the main-memory to the corresponding slot in theactive disk index segment, based on the keyword hash value.The third step then merges the main-memory data list, L,into the existing microblogs list on the disk. L data is firstchecked if it spans two days so that the list could be dividedinto two sublists and two index segments may be accessed.After that, the list (sublist) of microblogs is merged into thecorresponding slot by prepending the list to the existing disklist. This is O(1) operation due to the temporal order anddisjointness of the two lists. On the contrary, to consolidatedata from main-memory pyramid index to disk R+-tree in-dex, the data are flushed in raw format, where batches ofmicroblogs are bulk loaded to the R+-tree without any map-ping between the memory index partitions and disk indexpartitions. Although this builds the index partitions fromscratch, it is necessary due to the difference between theemployed spatial structures. As R+-tree is disk-friendly, thebulk loading flushing is efficient enough to handle the seg-mented microblogs data.

4.3 Flushing ManagementThe main task of Taghreed flushing manager is to deter-

mine which microblogs should be flushed from the main-memory indexes to disk indexes when the memory becomesfull. Although the task looks trivial for the first glance, ithighly affects the query performance [33] as it controls themain-memory contents. The incoming queries to Taghreedare answered from both main-memory and disk contents.The more relevant data in main-memory, the less disk accessis needed to answer the queries, and then the lower query re-sponse time. Thus, Taghreed flushing manager enables thesystem administrator to employ one of multiple availableflushing policies. The flushing policy tries to compromise theindexing and flushing overhead with the availability of rele-vant data, to incoming queries, in the main-memory. In this

section, we describe three different flushing policies namelyFlush-All, Flush-Temporal, and Flush-Query-Based.We also discuss the pros and cons of each of them.

Flush-All. The simplest flushing policy is to dump thewhole memory contents to the disk. This makes the main-memory indexing very flexible as any number of segmentscan be used without dramatic effect on the flushing process.Also, it minimizes the disk access overhead as less number offlushing operations are performed. Obviously, Flush-All pre-serves the property of temporal disjointness between main-memory contents and disk-contents as it always dump all theold data to the disk before receiving new recent data in themain-memory. However, Flush-All is not a recommendedflushing policy as it causes system slowdowns in terms ofquery responses. Once the flushing operation is performed,all the memory-indexes become empty and hence all the in-coming queries are answered only from disk contents. Thiscauses a significant sudden slowdown which is an undesirableeffect from a system point of view.

Flush-Temporal. An alternative flushing policy is toexpel a certain portion of the oldest microblogs to emptiesa room for the newly real-time incoming microblogs. Toreduce the flushing overhead, this policy requires the main-memory indexing to partition the data into segments withthe same flushing unit. Referring to the main-memory in-dex organization in Figure 2, the flushing unit is defined asT hours, i.e., the oldest T hours of data are flushed periodi-cally. In this policy, T would be a system parameter that isadjusted by the system administrator based on the availablememory resources, the rate of incoming microblogs, and thedesired frequency of flushing. Flush-Temporal also preservesthe property of temporal disjointness between main-memorycontents and disk-contents as it always dump data of a cer-tain period of time. This moves the temporal check pointtcp by exactly T hours without causing any kind of tempo-ral overlap. In addition, Flush-Temporal addresses the lim-itations of Flush-All and does not cause sudden significantsystem slowdowns. This comes on the cost of more frequentflushing operations which needs more frequent disk accessoverhead. However, as a background process, flushing diskaccess does not significantly affect the system performance.

Flush-Query-Based. A third policy is to expel the mi-croblogs that are not relevant to the incoming queries. Thisis important when it is required to optimize the system in-dexes to support certain query, or set of queries, efficiently.The idea is to figure out the characteristic of data that willnot satisfy the target query answer, and hence expel them.For example, if the query asks for most recent k microblogsthat contains a certain keyword. Then, if the inverted indexslot of any keyword contains more than k microblogs, thatmeans all microblogs older than the kth one will not make itto any query answer. Thus, those microblogs can be safelyexpelled to empties a space for more relevant microblogs toreside in the main-memory.

A fundamental problem in such kind of flushing policiesis how to preserve the property of the temporal disjointnessbetween main-memory contents and disk-contents. In thiscase, the microblogs are not expelled on either a tempo-ral criteria or from a contiguous temporal period. On thecontrary, they are expelled selectively based on the queryselection criteria. Thus, by default, there is no guaranteeabout the temporal overlap between the memory and diskcontents. The proposed solution in Taghreed system is to

168

flush the data to an intermediate disk buffer rather thanflushing directly to the disk indexes. Then, the query-basedflushing policy would be combined with a temporal flush-ing policy (with larger values of T ) so that after certainpoint of time it is guaranteed that all main-memory dataare more recent than a certain timestamp. At this point,all the data in the intermediate buffer could be merged tothe disk indexes without violating the temporal disjointness.This kind of policies would help to reduce the disk accessoverhead during the query processing. However, the inter-mediate disk buffer introduce difficulties of maintaining thebuffer and handle the temporal overlap gray area which isthe buffer data. Currently, Taghreed does not implementany query-based flushing policies. However, we have rigidplans to adapt some of these policies for top-k queries toprovide a system administration flexibility to tune the sys-tem performance for important queries. A detailed examplefor query-based flushing for top-k queries can be reviewed inour full paper on spatial search queries on microblogs [24].It is worth noting that all the flushing operations does

not affect the data availability. The main-memory data stayavailable to the incoming queries until the flushing opera-tion is successfully completed. Then, the temporal checkpoint, tcp, is atomically updated to indicate the new tempo-ral boundaries between the main-memory contents and thedisk contents. If concurrent queries have already read theold tcp value, the system keeps track of them using a simplepin counting technique before the flushing manager discardsthe flushed main-memory contents.

5. QUERY ENGINEThe second major component in Taghreed is the query

engine. This component consists of two main modules: aquery optimizer and a query processor. The two modulesare discussed in Sections 5.1 and 5.2, respectively.

5.1 Query OptimizerAs Section 4 shows, Taghreed provides two types of in-

dexes in both main-memory and disk: keyword index andspatial index. In addition, disk indexes data is replicatedon three temporal levels, daily, weekly, and monthly indexsegments. Consequently, the query processor may have dif-ferent ways to process the same query based on: (1) the or-der of performing keyword or spatial filtering based on thesystem indexes, and (2) the number of hit disk indexes. Forexample, the query that asks about only spatial data of theperiod from June 1 to June 9 can be answered from disk spa-tial indexes in two different ways: (a) either accessing ninedaily index segments, or (b) accessing one weekly and twodaily index segments. Each of those is called a query plan.The costs of different query plans are different. The maintask of the query optimizer is to generate a plan to executeso that the estimated cost is minimal. In this section, wediscuss Taghreed query optimization. First, we discuss thekeyword/spatial index order selection of both main-memoryand disk. Then, we discuss the disk index segments selec-tion over the three levels of temporal hierarchy. Finally, wedescribe the whole query plan generation scenario.Keyword/spatial index order selection. When a

query involves querying both spatial and keyword dimen-sions, there are two ways to retrieve the microblogs withinthe query scope: (1) either hit the keyword index and per-form spatial filtering for the retrieved microblogs, or (2) hit

the spatial index and perform keyword filtering for the re-trieved microblogs. To select one of the two plans, Taghreedquery optimizer employs a cost model for each scenario. Itcalculates the estimated cost for each plan and select thecheap one. For a query q, the costs of both plans are calcu-lated based on the following equations:

Cost(keyword|q) = Akw × query keyword count (1)

Cost(spatial|q) = Asp × query area (2)

Equation 1 is used to estimate the cost of hitting the key-word index given q while Equation 2 is used to estimatethe cost of hitting the spatial index given q. The cost of qdepends on its number of keywords and its spatial extent.Taghreed calculates a single number for each index, namely,Akw and Asp. Akw is the average number of microblogsin a keyword slot. Asp is the average number of processedmicroblogs per query area of one mile square. Using Akw

and Asp, the query optimizer is able to estimate the num-ber of microblogs need to be processed to provide the queryanswer. In main-memory, this estimates the amount of pro-cessing needed. On disk, this estimates the number of pagesneeded to be retrieved from disk. Next, we discuss the cal-culation of Akw and Asp.

To calculate Akw, two numbers are maintained for eachkeyword index: the total number of microblogs inserted sofar in the index TotalM , and the number of distinct key-words inserted in the index Nkw. Akw can be then calcu-lated as follows: Akw = TotalM

Nkw. Both TotalM and Nkw are

easy to maintain during the index update operations withalmost no overhead. To calculate Asp, two numbers aremaintained for each spatial index: the summation of aver-age numbers of microblogs processed under each incomingquery since the index is created Sumavg, and the number ofqueries processed on the index Nq. Asp can be then calcu-

lated as follows: Asp =Sumavg

Nq. To maintain Sumavg, for

each query, Taghreed keeps track of the total number of pro-cessed microblogs during the query. Then, this number isdivided by the query area (in miles square). Finally, the di-vision result is added to Sumavg while Nq is incremented byone. It is worth noting that Akw is changing over time withthe data keyword distribution while Asp is changing overtime with the query load spatial distribution. This dynamiclearning process of Akw and Asp continuously improves thecost estimation and hence the query performance.

Disk index segments selection. As data on disk isreplicated on three levels of temporal hierarchy, there areusually different ways to access data in certain temporalrange: from either a daily index, a weekly index, or amonthly index. To estimate the cost of hitting each indexindividually, we use Equations 1 and 2. However, there aredifferent valid combinations of indexes. Trying to minimizethe overhead of getting an optimal combination of indexes,the query optimizer starts with the combination with theminimum amount of data to be accessed. This means us-ing weekly and monthly indexes only for whole weeks andwhole months, respectively, in the query temporal horizon.For example, if the query temporal horizon spans May 29 toJuly 9, then the starting combination would be three dailyindexes for last three days of May, one monthly index forwhole June, one weekly index for first week of July, and

169

two daily indexes for July 8 and 9. As these indexes donot contain any data outside the query temporal boundary,so, it contains the minimum amount of data to be accessed.The next step is to figure out the best combination. Theassumption made here that going up in the index temporalhierarchy would increase the cost. In our example, replacingthe three daily indexes of the last three days of May with oneweekly index of the last week of May would incur more costas more disk pages would be retrieved. Thus, the employedheuristic is to go down in the index hierarchy to explore anddivide weekly and monthly indexes into finer granularity in-dexes, i.e., days and weeks, respectively. Starting from thefirst generated combination, the query optimizer tries to re-place weekly indexes with seven daily indexes (and monthlyindexes with four weekly indexes). Checking the costs of thiscombinations are not costly as it is just summation of seven(four) cost parameters numbers, i.e., Akw and Asp. Theoptimizer then selects the combination with the minimumestimated cost.Query plan generation. To generate a complete query

plan, first the optimizer checks the query temporal horizonversus the memory/disk data temporal boundary, tcp, to de-termine the temporal horizon of both memory and disk data,namely, tm and td, respectively. Afterwards, a sub-plan isgenerated for each of them separately. In main-memory,the index segments that intersect with tm are determined.Then, the index to be hit (either keyword or spatial) is se-lected based on the above selection model. On disk, an indexcombination is generated for both spatial and keyword indexhierarchies. Then, the cost of each combination is estimatedbased on Equations 1 and 2 and the cheapest one is selected.

5.2 Query ProcessorTo provide flexible and efficient spatio-temporal query-

ing framework, Taghreed has chosen to employ indexes onspatial, temporal, and keyword attributes and perform fil-tering on all other attributes through efficient distributeddata scanners (see Section 4 for more details on indexingattributes selection). Thus, Taghreed query processor hastwo phases of answering any queries: (1) retrieving a candi-date set of microblogs from a spatio-temporal or a keyword-temporal space, depending on the query plan, and (2) per-forming further processing through scanning on the candi-date set, if needed. We discuss the two phases below.Phase 1: Filtering on the spatio-temporal keyword

attributes. In this phase, the query processor retrievesa list of candidate microblogs based on the query spatial,temporal, and keyword parameters. This is performed byexecuting the optimized query plan (that is generated asdescribed in Section 5.1) through hitting the system indexes.Specifically, the query processor receives a query plan thatconsists of an optimized set of indexes to be accessed. Eachof the indexes is queried to retrieve a list of microblogs thatsatisfy the user query parameters. The candidate lists arethen fed to the second phase for further refinement. As theindexes provide efficient pruning on the indexed attributes,this phase prunes a huge amount of data.Phase 2: Refinement on other attributes. The out-

put of the phase 1 would be lists of microblogs that requirefurther processing. This phase performs the remaining pro-cessing, through extensive distributed data scanning, to pro-vide the final query answer. The type of processing dependson the query type and the query plan. If the spatial index is

hit in phase 1, then keyword filtering would be performed inphase 2 and vice versa. Also, this scanning is piggybacked byother operations on any other attributes, e.g., counting mi-croblogs of distinct languages or count frequent keywords,so that Taghreed can support a wide variety of queries ondifferent attributes. More details and examples about thesupported queries are presented in Section 7.

6. RECOVERY MANAGERWith hours and even days of data managed in main-

memory, Taghreed system accounts for any failures that maylead to data loss. Taghreed employs a simple, yet effective,triple-redundancy model where the main-memory data isreplicated three times over different machines. The core ideaof this model is similar to Hadoop redundancy model thatreplicates the data three times. In this section, we brieflydescribe the operation of the recovery management modulein Taghreed in both normal and failure cases.

When Taghreed is launched, all the main-memory mod-ules, e.g., indexes and all data structures, are initiated onthree different machines. Each machine is fed with ex-actly the same stream of microblogs, thus they form tripleidentical copies of the main-memory system status. Oneof the three machines is a master machine that launchesall the system components, i.e., memory-resident and disk-resident components. The other two machines launch onlythe memory-resident components. Any flushing from mem-ory to disk in the master machine leads to throwing the dataout from the memory of the other two machines. On failureof the master machine, the other two machines continue todigest the real-time microblogs. Once the master machine isrecovered, the system memory image is copied to its main-memory from one of the other machines. On the failure ofone of the secondary machines, the other machine data isused to create an alternative for the failed machine. Repli-cating the data three times significantly reduces the proba-bility of having the three machines down simultaneously andlose all the main-memory data.

Although Taghreed recovery management is not 100%strict, it is acceptable for the relaxed social media appli-cations that favor efficiency and scalability over strict con-sistency and recovery management. Recovery managementthrough main-memory replication allow these applicationsto scale without being limited with the overhead of disk-based recovery interactions. In addition, its cost is almostdoubling or tripling the cost of the needed main-memory re-sources which is affordable in the age when the memory costis becoming lower over time.

7. VISUALIZERTaghreed is a full-fledged system that provides end-to-

end solution for microblogs users. Thus, Taghreed providesan interactive visualizer component that interacts with endusers. Such component is very important for designing arich set of interactive queries along with their friendly userinterfaces. As a flexible querying framework for microblogsdata, Taghreed core data management components are ableto support a wide variety of queries efficiently which enablessuch richness of query design. In this section, we presentan overview about the visualizer component and its integra-tion with the back end components. Then, we present thecurrently supported queries along with system interfaces tovisualize these queries to end users.

Visualizer overview. Taghreed interactive visualizer is

170

Figure 4: Taghreed Integrated Interface.

the system front end. It receives user queries through in-teractive web-based user interfaces. The queries are thendispatched to the query engine through Java-based functionAPIs that allow fast interaction, eliminating the overhead ofexchanging data in standard formats, e.g., JSON or XML.The query processor then sends back the queries answers sothat the visualizer present them to end users.Supported queries. As discussed in Section 3.3,

Taghreed system is designed to provide a flexible frameworkthat is able to answer a wide variety of spatio-temporalqueries on different microblogs attributes. Thus, in thispart, we present only a sample of queries that are currentlysupported. However, Taghreed is not limited to answerthe discussed queries and can be adapted to answer morespatio-temporal keyword queries. Currently, Taghreed sup-ports the following spatio-temporal queries on microblogs:

1. Keyword search. Within given spatial and temporalranges, find all microblogs that contain certain key-words.

2. Top-k frequent keywords. Within given spatialand temporal ranges, find the most frequent k key-words, for a given integer k.

3. Top-k active users. Within given spatial andtemporal ranges, find the most k active users, for agiven integer k. Active users are defined as the userswho have posted the largest number of microblogs inthe query spatio-temporal range.

4. Top-k famous users. Within given spatial andtemporal ranges, find the most k famous users, for agiven integer k. Famous users are defined as the usershaving the largest number of followers. The queryanswer is selected from users whose home locationslie in the query spatial range and have posted at leastone microblog during the query temporal range.

5. Daily aggregates. Within given spatial and tempo-ral ranges, find the number of microblogs in each day.

6. Joint collective queries. Within given spatial andtemporal ranges, find the answer of all the previousqueries collectively.

7. Top-k used languages. Within given spatial andtemporal ranges, find the most k used languages, for agiven integer k.

It is worth noting that there are a variety of queries thatrequire scanning on attributes other than spatial, tempo-ral, and keyword. All such scanning efforts are piggybackedon phase 2 of the query processor (see Section 5.2). Also,query 6 is an example of collective queries where multiplequeries share the processing power. This significantly re-duces the amount of processing consumed per microblog.

System interfaces. Currently Taghreed provides twouser interfaces, one integrated interface to visualize queries 1to 6 in addition to a separate interface that employs query 7in a different way than just presenting a result of userinput query. Figure 4 shows the main integrated inter-face. Through this interface, Taghreed user can inputa spatial range (through the map interface), a temporalrange (through the datepicker), and an optional keywords(through text box). The system then dispatches the collec-tive query 6 with default k=10 and present to the resultsto the user in the five side boxes in Figure 4. Figure 5shows another interface that employs query 7 to provide ananalysis for language usage in Arab Gulf area using Twitterdata. The query is issued for all the sub-regions, then theoutput pie charts are displayed on the map interface. Thegranularity of the results change at different zoom levels.

8. CONCLUSIONThis paper presented Taghreed; a system for scal-

able querying and visualization of geotagged microblogs.Taghreed is able to manage and query Billions of microblogsthrough four main components. First, an indexer that han-dles recent microblogs in segmented main-memory indexeswith their high arrival rates. When memory becomes full,certain microblogs, selected through the flushing manager,

171

Figure 5: Language Distribution Query in GulfArea.

are expelled to disk-resident indexes. Second, a query en-gine that provides query optimization and a query process-ing on top of system indexes. Third, a recovery managerthat restores the system status in case of main-memory fail-ure. Fourth, an interactive visualizer that interacts withsystem end users. Taghreed provides a flexible frameworkthat can adapt several queries that involve spatial, tempo-ral, and keyword attributes.

9. REFERENCES[1] H. Abdelhaq, C. Sengstock, and M. Gertz. EvenTweet: Online

Localized Event Detection from Twitter. In VLDB, 2013.

[2] S. Alsubaiee, Y. Altowim, H. Altwaijry, A. Behm, V. R. Borkar,Y. Bu, M. J. Carey, R. Grover, Z. Heilbron, Y.-S. Kim, C. Li,N. Onose, P. Pirzadeh, R. Vernica, and J. Wen. ASTERIX: AnOpen Source System for “Big Data“ Management and Analysis.PVLDB, 5(12):1898–1901, 2012.

[3] Apple buys social media analytics firm Topsy Labs.http://www.bbc.co.uk/news/business-25195534, 2013.

[4] W. G. Aref and H. Samet. Efficient Processing of WindowQueries in the Pyramid Data Structure. In PODS, 1990.

[5] A. Bermingham and A. F. Smeaton. Classifying Sentiment inMicroblogs: Is Brevity an Advantage? In CIKM, 2010.

[6] After Boston Explosions, People Rush to Twitter for BreakingNews. 2013. http://www.latimes.com/business/technology/la-fi-tn-after-boston-explosions-people-rush-to-twitter-for-breaking-news-20130415,0,3729783.story.

[7] M. Busch, K. Gade, B. Larson, P. Lok, S. Luckenbill, andJ. Lin. Earlybird: Real-Time Search at Twitter. In ICDE, 2012.

[8] C. C. Cao, J. She, Y. Tong, and L. Chen. Whom to Ask? JurySelection for Decision Making Tasks on Micro-blog Services.PVLDB, 5(11), 2012.

[9] C. Chen, F. Li, B. C. Ooi, and S. Wu. TI:An Efficient IndexingMechanism for Real-Time Search on Tweets. In SIGMOD,2011.

[10] L. Chen, G. Cong, C. S. Jensen, and D. Wu. Spatial KeywordQuery Processing: An Experimental Evaluation. In VLDB,2013.

[11] Sina Weibo, ChinaSs Twitter, comes to rescue amid flooding inBeijing. 2012. http://thenextweb.com/asia/2012/07/23/sina-weibo-chinas-twitter-comes-to-rescue-amid-flooding-in-beijing/.

[12] A. Dong, R. Zhang, P. Kolari, J. Bai, F. Diaz, Y. Chang,Z. Zheng, and H. Zha. Time is of the essence: Improvingrecency ranking using twitter data. In WWW, 2010.

[13] Facebook Statistics.http://newsroom.fb.com/Key-Facts/Statistics-8b.aspx, 2012.

[14] R. A. Finkel and J. L. Bentley. Quad Trees: A Data Structurefor Retrieval on Composite Keys. ACTA, 4(1), 1974.

[15] J. Hannon, M. Bennett, and B. Smyth. Recommending twitterusers to follow using content and collaborative filteringapproaches. In RecSys, 2010.

[16] J. Hart, C. Ridley, F. Taher, C. Sas, and A. Dix. Exploring theFacebook Experience: A New Approach to Usability. InProceedings of the 5th Nordic Conference onHuman-computer Interaction, NordiCHI, pages 471–474, 2008.

[17] Harvard Tweet Map. http://worldmap.harvard.edu/tweetmap/,2013.

[18] S. J. Kazemitabar, U. Demiryurek, M. H. Ali, A. Akdogan, andC. Shahabi. Geospatial Stream Query Processing usingMicrosoft SQL Server StreamInsight. PVLDB, 3(2), 2010.

[19] G. Lee, J. Lin, C. Liu, A. Lorek, and D. V. Ryaboy. TheUnified Logging Infrastructure for Data Analytics at Twitter.PVLDB, 5(12), 2012.

[20] R. Li, K. H. Lei, R. Khadiwala, and K. C.-C. Chang. TEDAS:A Twitter-based Event Detection and Analysis System. InICDE, 2012.

[21] J. Lin and G. Mishne. A Study of ”Churn” in Tweets andReal-Time Search Queries. In ICWSM, 2012.

[22] W. Liu, Y. Zheng, S. Chawla, J. Yuan, and X. Xing.Discovering Spatio-temporal Causal Interactions in Traffic DataStreams. In KDD, 2011.

[23] A. Magdy, A. M. Aly, M. F. Mokbel, S. Elnikety, Y. He, andS. Nath. Mars: Real-time Spatio-temporal Queries onMicroblogs. In ICDE, pages 1238–1241, 2014.

[24] A. Magdy, M. F. Mokbel, S. Elnikety, S. Nath, and Y. He.Mercury: A Memory-Constrained Spatio-temporal Real-timeSearch on Microblogs. In ICDE, pages 172–183, 2014.

[25] map-D: Massively Parallel Database.http://mapd.csail.mit.edu/, 2014.

[26] A. Marcus, M. S. Bernstein, O. Badar, D. R. Karger,S. Madden, and R. C. Miller. Tweets as Data: Demonstrationof TweeQL and TwitInfo. In SIGMOD, 2011.

[27] A. Marcus, M. S. Bernstein, O. Badar, D. R. Karger,S. Madden, and R. C. Miller. Twitinfo: Aggregating andVisualizing Microblogs for Event Exploration. In CHI, 2011.

[28] D. McCullough, J. Lin, C. Macdonald, I. Ounis, and R. M. C.McCreadie. Evaluating Real-Time Search over Tweets. InICWSM, 2012.

[29] E. Meij, W. Weerkamp, and M. de Rijke. Adding semantics tomicroblog posts. In WSDM, 2012.

[30] E. Meskovic, Z. Galic, and M. Baranovic. Managing MovingObjects in Spatio-temporal Data Streams. In MDM, 2011.

[31] G. Mishne and J. Lin. Twanchor Text: A Preliminary Study ofthe Value of Tweets as Anchor Text. In SIGIR, 2012.

[32] M. F. Mokbel and W. G. Aref. SOLE: Scalable On-LineExecution of Continuous Queries on Spatio-temporal DataStreams. VLDB Journal, 17(5), 2008.

[33] M. F. Mokbel, M. Lu, and W. G. Aref. Hash-Merge Join: ANon-blocking Join Algorithm for Producing Fast and EarlyJoin Results. In ICDE, pages 251–262, 2004.

[34] O. Phelan, K. McCarthy, and B. Smyth. Using twitter torecommend real-time topical news. In RecSys, 2009.

[35] D. Ramage, S. T. Dumais, and D. J. Liebling. CharacterizingMicroblogs with Topic Models. In ICWSM, 2010.

[36] T. Sakaki, M. Okazaki, and Y. Matsuo. Earthquake shakestwitter users: Real-time event detection by social sensors. InWWW, 2010.

[37] J. Sankaranarayanan, H. Samet, B. E. Teitler, M. D.Lieberman, and J. Sperling. TwitterStand: News in Tweets. InGIS, 2009.

[38] Topsy Pro Analytics: Find the insights that matter.http://topsy.com/, 2013.

[39] TweetTracker: track, analyze, and understand activity onTwitter. http://tweettracker.fulton.asu.edu/, 2013.

[40] Twitter Data Grants, 2014.https://blog.twitter.com/2014/introducing-twitter-data-grants.

[41] New features on Twitter for Windows Phone 3.0, 2013.https://blog.twitter.com/2013/new-features-on-twitter-for-windows-phone-30.

[42] Twitter Statistics.http://business.twitter.com/en/basics/what-is-twitter/, 2013.

[43] I. Uysal and W. B. Croft. User Oriented Tweet Ranking: AFiltering Approach to Microblogs. In CIKM, 2011.

[44] K. Watanabe, M. Ochi, M. Okabe, and R. Onai. Jasmine: AReal-time Local-event Detection System based on GeolocationInformation Propagated to Microblogs. In CIKM, 2011.

[45] L. Wu, W. Lin, X. Xiao, and Y. Xu. LSII: An IndexingStructure for Exact Real-Time Search on Microblogs. In ICDE,2013.

[46] J. Yao, B. Cui, Z. Xue, and Q. Liu. Provenance-based IndexingSupport in Micro-blog Platforms. In ICDE, 2012.

[47] D. Zhang, D. Gunopulos, V. J. Tsotras, and B. Seeger.Temporal and Spatio-temporal Aggregations over Data Streamsusing Multiple Time Granularities. Information Systems,28(1-2), 2003.

172