Hortonworks HBase Meetup Presentation

© Hortonworks Inc. 2011

HBase Dev Meetup 09/11/2012

Enis Soztutar

enis [@] apache [dot] org

#enissoz

Page 1


Integration Testing

• Hbase Integration Tests plan– HBASE-6201

1. Create hbase-it module

2. Ability to run integration tests on a given clustermvn verifybin/hbase org.apache.hadoop.hbase.IntegrationTestsDriver

3. Port candidate tests

4. Add more long running and fault injection tests

5. (Most of) Integration tests should be able to run over a mini cluster or a 100+ node cluster.

6. Integration tests should run in mini mode as nightlies (apache jenkins)

7. Our plan is to run the whole suite of Integration tests nightly/weekly on medium-sized clusters. Every org. should be able to do this easily.

Page 2Architecting the Future of Big Data


Current Status

• hbase-it module committed, HBASE-6203• HBaseCluster interface for interacting with the cluster from system tests

(HBASE-6241)– Patch near to being committed

• Subtask jiras for porting each test or adding a new test• Hbase book documentation available in patch for HBASE-6302• Candidate tests to be converted:


• LoadTestTool• BigTop TestLoadAndVerify• Goraci (https://github.com/keith-

turner/goraci)• TestAcidGuarantees /

TestAtomicOperation• TestRegionBalancing (HBASE-

6053)• TestFullLogReconstruction• TestMultiVersions /

TestKeepDeletes

• TestFromClientSide• TestShell and src/test/ruby• TestRollingRestart• TestMasterFailover• TestImportExport• Test**OnCluster• Balancer tests


HBASE-6241 Overview

• New junit @Category(IntegrationTests.class)• Introduces HBaseCluster

– public abstract ClusterStatus getClusterStatus();– public abstract void startRegionServer(String hostname);– Start/stop/kill RS / Master– MiniHBaseCluster and DistributedHBaseCluster implements HBaseCluster

• Introduces DistributedHBaseCluster– Keeps state about a distributed cluster – Does not know about environment. Uses ClusterManager to interact with the

cluster

• ClusterManager is an Api to manage services– public abstract void start(ServiceType service, String hostname)– Pluggable, specific implementations for every environment

• HBaseClusterManager extends ClusterManager– Knows about environment (HBASE_HOME, HBASE_CONF_DIR,

HBASE_SSH_OPTS )– Uses SSH and remotely executes ps, kill -9, bin/hbase-daemon.sh start/stop



HBASE-6241 Overview (cont.)

• Introduces HBaseIntegrationTestingUtility extends HBaseTestingUtility– Integration tests uses this

• Introduces 2 ways to run integration tests – bin/hbase o.a.h.h.IntegrationTestsDriver –config <hbase_conf_dir>

– mvn verify (runs tests using mini cluster)– mvn verify –Dhbase.test.cluster.distributed=true –Dhbase.conf.dir=<hbase_conf_dir> (further patch)

• Introduces ChaosMonkey– A utility to injects faults in a running cluster.– Does not know about environment, uses HBaseIntegrationTestingUtility– Actions (RestartActiveMaster, RestartRandomRs, RestartRsHoldingMeta,

BatchRestartRs, RollingBatchRestartRs, etc )– Policies (PeriodicRandomActionPolicy), predefined named policies

(EVERY_MINUTE_RANDOM_ACTION_POLICY)

• Introduces IntegrationTestDataIngestWithChaosMonkey– Runs LoadTestTool and ChaosMonkey for configurable running time– Adjusts data load based on the cluster size (also works with mini cluster)



Open Issues

•0.94 porting?•Hadoop NN HA testing•BIGTOP-635•Community interest•Different environments (no SSH, etc)•???



HBase on Windows

• WHY? –Because we can : )–More platforms = more users + more development

• WHO–MS + HW

• WHEN?–Patches will start rolling shortly

• WHERE?–Apache HBase Trunk–0.94? 0.92?

• WHAT– Hadoop on windows (HADOOP-8079)– Test fixes, file path name related fixes, timing issues, network related changes,

build changes, bin scripts, etc



ThanksGo bananas!

Architecting the Future of Big DataPage 8

Taken from http://99designs.com/t-shirt-design/contests/design-chaos-monkey-t-shirt-70909/entries?entriespage=1#contest-header

Technology

Hortonworks HBase Meetup Presentation