Secondary Indexing in Phoenix - Hadoop Summit 2012 - HBase BoF

Secondary Indexing in Phoenix

Jesse YatesHBase CommitterSoftware Engineer

HBase BoF – June 25, 2013

HBase BoF - June 20132

Outline

• Motivation• History• HBase Consistent Indexing– Index Management– Recovery Mechanism

• Conclusion


A quick note…


Outline


• Conclusion


Why do we need them?

• Sorted by key– Great for accessing on that key

What if we want to access by another dimension!?Table Scan!


A short example

• Easy to search by name of food

• Hard to search on another dimension

Name Type Date Received Manufacturer Current Count

Apple Macintosh 6/23/13 Good Farm Inc. 200

Turkey Breast 6/23/13 Tasty Meat Co. 42

Chicken Drumstick 6/18/13 Pretty Ok Food 3

Jam Strawberry 6/18/10 Mash It Up Inc. 700


A short exampleName Type Date Received Manufacturer Current Count

Apple Macintosh 6/23/13 Good Farm Inc. 200

Turkey Breast 6/23/13 Tasty Meat Co. 42

Chicken Drumstick 6/18/13 Pretty Ok Food 3

Jam Strawberry 6/18/10 Mash It Up Inc. 700

Date Received Name Type Manufacturer Current Count

6/18/13 Jam Strawberry Mash It Up Inc. 700

6/18/13 Chicken Drumstick Pretty Ok Food 3

6/23/13 Apple Macintosh Good Farm Inc. 200

6/23/13 Turkey Breast Tasty Meat Co. 42


Outline


• Conclusion


HBase is “Special”…

• Partitioned Keys (“HRegion”)

• Scales because regions are independent

• Built-in data recovery mechanisms


Hasn’t someone tried this?

• Omid

• Percolator

• Culvert

• Lily

• TrendMicro

• Client-coordinated


We’ve gotten better…

• NGData– HBase-SEP– HBase-Indexer

• Intel– Lucene Full Text Indexing


Still missing some things

• In-HBase index storage– Just another table in HBase

• Simple consistency guarantees– If X fails, then Y

• Minimal overhead for covered indexes– Network roundtrips


Outline


• Conclusion

14

Two Major Components

• Index Management– Build index updates– Ensures index is ‘cleaned up’

• Recovery Mechanism– Ensures index updates are “ACID”



Index Management

• Lives within a RegionCoprocesorObesrver• Access to the local Hregion• Specifies the mutations to apply to the index

tables

public interface IndexBuilder {public void setup(RegionCoprocessorEnvironment env);public Map<Mutation, String> getIndexUpdate(Put put);public Map<Mutation, String> getIndexUpdate(Delete delete);

}


Index Management

Key Observation #1

“We shouldn’t need to provide stronger guarantees than HBase - that is just asking for a bad time.”

- Jon Hsieh


* Paraphrased

*


HBase ACID

• Does NOT give you:– Cross-row consistency– Cross-table consistency

• Does give you:– Durable data on success– Visibility on success without partial rows

Key Observation #2

“Secondary indexing is inherently an easier problem than full transactions… secondary index updates are idempotent.”

- Lars Hofhansl



Idempotent Index Updates

• Doesn’t need full transactions

• Replay as many times as needed

• Can tolerate a little lag– As long as we get the order right

Taking a little ACID…




Durable Indexing: Standard Write Path

Client HRegion

RegionCoprocessorHost

WAL


MemStore


Durable Indexing: Standard Write Path

Client HRegion


WAL


MemStore


Durable Indexing

RegionCoprocessor

Host

WAL


Indexer IndexBuilder

WAL Updater

Durable!

IndexerIndex Table

Index TableIndex Table


Failure Situations

• Before writing the WAL– Nothing is durable, nothing is visible


Durable Indexing

Client HRegion


WAL


MemStore

Indexer

Indexer

Index TableIndex TableIndex Table


Failure Situations


✔


Failure Situations


• After writing WAL, before index update– WAL Replay updates the index table and the

primary table

✔


Durable Indexing

Client HRegion


WAL


MemStore

Indexer

Indexer



Failure Situations



primary table

✔

✔


Failure Situations



primary table• Mid-index update– WAL Replay finishes index update, primary table

update

✔

✔


Durable Indexing

Client HRegion


WAL


MemStore

Indexer

Indexer



Failure Situations• Before writing the WAL– Nothing is durable, nothing is visible

• After writing WAL, before index update– WAL Replay updates the index table and the primary

table• Mid-index update– WAL Replay finishes index update, primary table

update

✔

✔

✔


Failure Situations



table• Mid-index update– WAL Replay finishes index update, primary table update

• After index updates, before primary– WAL Replay restores primary state, idempotently

applies index updates

✔

✔

✔


Durable Indexing

Client HRegion


WAL


MemStore

Indexer

Indexer



Failure Situations



table• Mid-index update– WAL Replay finishes index update, primary table update

• After index updates, before primary– WAL Replay restores primary state, idempotently

applies index updates

✔

✔

✔

✔


Special Note: Failed Index Updates

• Index is corrupted– Index Table does not exist– Index table does not have write schema– Etc.

• Fail-fast behavior– Kill the whole server– Forces WAL Replay to enforce correctness– Modular enough to support alternative schemes


Key Points

• Custom KeyValues to enable index durability in primary table WAL

• Custom WALEdit Codec for index update with WAL Replay

• Will see index updates before primary– Only a little bit of lag and never ‘wrong’– Matches HBase consistency

• Fail-fast behavior to enforce correctness


Upcoming Work

• Performance testing

• Standard covered index managers

• Index cleanup on compaction


Outline


• Conclusion


Conclusion

• Fully transparent to client

• Easy to build custom index maintenance

• Meets current HBase consistency guarantees

• Supports HBase 0.94.9+– Coming to 0.96/0.98 soon!

hbase-index


https://github.com/forcedotcom/phoenix/tree/master/contrib/hbase-index

Detailed Blog Post


http://jyates.github.io/2013/06/11/hbase-consistent-secondary-indexing.html


Bonus!

• Usable as a standalone module

• Coming to phoenix*– Built-in support

• Future: added to HBase core (?)

* https://github.com/forcedotcom/phoenix

Thanks! Questions!


@[email protected]

Business

Secondary Indexing in Phoenix - Hadoop Summit 2012 - HBase BoF