10
Stacki Lab Joe Kaiser Director of Open Source Engineering

Building a Hadoop Cluster with Stacki

  • Upload
    stackiq

  • View
    121

  • Download
    1

Embed Size (px)

Citation preview

Page 1: Building a Hadoop Cluster with Stacki

Stacki Lab

Joe Kaiser

Director of Open Source Engineering

Page 2: Building a Hadoop Cluster with Stacki

Open Source Stack Installer

Stacki is a very fast and ultra reliable Linux server provisioning tool … at scale.

With zero prerequisites for taking systems from bare metal to a ping and prompt.

Page 3: Building a Hadoop Cluster with Stacki

Stuff it does

Installs to bare metal or VMs that look like bare metal

Kickstart based

Parallel formatting of disk

Parallel Sharing of RPMs

CentOS/RHEL

Networking

Multiple subnets, vlaning, bonding.

Storage

Controller config

Partitioning

Whatever you can do in Linux, you can do with Stacki only clustered

Page 4: Building a Hadoop Cluster with Stacki

Stacki and Hortonworks Data Platform

Page 5: Building a Hadoop Cluster with Stacki

Hortonworks – stacki-hdp-bridge pallet

Add/enable/run

Stacki creates ambari appliance

Download software as pallets (isos)

Add/enable HDP and Ambari

Run gethdp script

Or download (links in documentation)

Assign a node to be Ambari deployment.

Do partitions and preps backend nodes for Hadoop.

Install all.

Go to Ambari interface to deploy Hadoop.

Current versions: (updated /export/HDP/hdp.cfg for new versions)

distribution = 2.x

os = centos7

ambari = 2.4.2.0

hdp = 2.5.3.0

Page 6: Building a Hadoop Cluster with Stacki

Augment the “default” Box

1) Replace the “os” pallet with the “CentOS” and “CentOS-

Updates” pallets

2) Add the Hortonworks pallets: “HDP”, “HDP-UTILS”,

“Updates-ambari” or run /export/HDP/gethdp.py

3) Add a pallet to glue the two layers together: “stacki-

hdp-bridge”

Page 7: Building a Hadoop Cluster with Stacki

Specify One Host as the “Ambari” appliance

“backend-0-0” will be automatically configured with Ambari

Install the node and then point your web browser at it

Page 8: Building a Hadoop Cluster with Stacki

Reinstall All Backend Nodes

Wipe all hardware disk array configuration and rebuild all the LUNs:

◦ stack set host attr ambari backend attr=nukecontroller value=true

Remove all partitions then repartition and reformat the disks:

◦ stack set host attr ambari backend attr=nukedisks value=true

Instruct nodes to install on next PXE boot:

◦ stack set host boot ambari backend action=install

Page 9: Building a Hadoop Cluster with Stacki

Try It

Website

www.stacki.com

Vagrant tire kick

https://github.com/rfkrocktk/vagrant-stacki

Source Code and docs

github.com/stackiq/stacki

github.com/StackIQ/stacki-hdp-bridge

Slack Channel (because everyone)

Google Groups

groups.google.com/forum/#!forum/stacki

Page 10: Building a Hadoop Cluster with Stacki

Finis

Thanks