Building a Hadoop Cluster with Stacki

Stacki Lab

Joe Kaiser

Director of Open Source Engineering

Open Source Stack Installer

Stacki is a very fast and ultra reliable Linux server provisioning tool … at scale.

With zero prerequisites for taking systems from bare metal to a ping and prompt.

Stuff it does

Installs to bare metal or VMs that look like bare metal

Kickstart based

Parallel formatting of disk

Parallel Sharing of RPMs

CentOS/RHEL

Networking

Multiple subnets, vlaning, bonding.

Storage

Controller config

Partitioning

Whatever you can do in Linux, you can do with Stacki only clustered

Stacki and Hortonworks Data Platform

Hortonworks – stacki-hdp-bridge pallet

Add/enable/run

Stacki creates ambari appliance

Download software as pallets (isos)

Add/enable HDP and Ambari

Run gethdp script

Or download (links in documentation)

Assign a node to be Ambari deployment.

Do partitions and preps backend nodes for Hadoop.

Install all.

Go to Ambari interface to deploy Hadoop.

Current versions: (updated /export/HDP/hdp.cfg for new versions)

distribution = 2.x

os = centos7

ambari = 2.4.2.0

hdp = 2.5.3.0

Augment the “default” Box

1) Replace the “os” pallet with the “CentOS” and “CentOS-

Updates” pallets

2) Add the Hortonworks pallets: “HDP”, “HDP-UTILS”,

“Updates-ambari” or run /export/HDP/gethdp.py

3) Add a pallet to glue the two layers together: “stacki-

hdp-bridge”

Specify One Host as the “Ambari” appliance

“backend-0-0” will be automatically configured with Ambari

Install the node and then point your web browser at it

Reinstall All Backend Nodes

Wipe all hardware disk array configuration and rebuild all the LUNs:

◦ stack set host attr ambari backend attr=nukecontroller value=true

Remove all partitions then repartition and reformat the disks:

◦ stack set host attr ambari backend attr=nukedisks value=true

Instruct nodes to install on next PXE boot:

◦ stack set host boot ambari backend action=install

Try It

Website

www.stacki.com

Vagrant tire kick

https://github.com/rfkrocktk/vagrant-stacki

Source Code and docs

github.com/stackiq/stacki

github.com/StackIQ/stacki-hdp-bridge

Slack Channel (because everyone)

Google Groups

groups.google.com/forum/#!forum/stacki

Finis

Thanks