© Hortonworks Inc. 2015

Hadoop and Kerberos: The madness beyond the gate

Steve Loughranstevel@hortonworks.com@steveloughran2015

Page 2

Me: Before Kerberos

© Hortonworks Inc.Page 3

Me: After Kerberos

© Hortonworks Inc.Page 4

HP Lovecraft Kerberos

Evil lurking in New England MIT Project Athena

Ancient, inhuman deities Kerberos Domain Controller

Manuscripts to drive the reader insane

IETF RFC 4120

Entities never spoken of aloud UserGroupInformation

Doomed explorers of darkness You

© Hortonworks Inc. 2015

Leave now if you want to retain your life ofnaïve innocence

Page 5

© Hortonworks Inc.Page 6

© Hortonworks Inc. 2015

export HADOOP_USER="root"

Page 7

© Hortonworks Inc. 2015

Modern Hadoop clusters are locked downthrough Kerberos

Page 8

© Hortonworks Inc. 2015

Discover Kerberosbefore Kerberosdiscovers you

Page 9

© Hortonworks Inc. 2015

Kerberos:the gateway to hell

Page 10

© Hortonworks Inc.

This is not a metaphor

Art: Andrés Álvarez Iglesias

© Hortonworks Inc. 2015

KP

Kerberos is the gateway

Page 12

Authentication Service

Ticket Granting Service

Principal

user@REALMuser/hostname@REALM

(P, TGS, n1)

{KP.TGS, n1}KP, {ticket(P,TGS)}

KTGS Ticket(P, TGS) = (TGS, P, tstart, tend, KPT)

KP

{KP.S, n2}KP, {ticket(P,S)} KS

{auth(P)}KP.TGS,{ticket(P,TGS)}KTGS,S,n2

KTGS

Kerberos Domain ControllerClient

auth(P)KP.TGS = {P, time)}KP.TGS

© Hortonworks Inc

Every service is a principal

alice@REALMbob@REALM

oozie/ooziehost@REALM

namenode/nn1@REALM

hdfs/_HOST@REALMhdfs/r04s12@REALMhdfs/r04s13@REALM

yarn/_HOST@REALMyarn/r04s12@REALM

HTTP/_HOST@REALMPage 13

short names:aliceboboozienamenodehdfsyarnHTTP

© Hortonworks Inc.Page 14

Entering the darkness

© Hortonworks Inc.Page 16

© Hortonworks Inc. 2015

HDFS Bootstrap: Kerberos Login

Page 17

shared keytab in /etc/hadoop

log in to kerberos

datanode/_HOST@REALM

tickets for TGS

namenode/nn@REALM

© Hortonworks Inc. 2015

HDFS Bootstrap: DNs register with NN

Page 18

shared keytab in /etc/hadoop

DN registration

Ticket for namenode/nn@REALM

ExportedBlockKeys

Request ticket for namenode/nn@REALM

namenode/nn@REALM

datanode/_HOST@REALM

© Hortonworks Inc.

Hadoop Tokens

• Issued and tracked by individual services

(HDFS, WebHDFS, Timeline Server, YARN RM, …)

• Grant some form of access:

Block tokens, Delegation Tokens

• Can be passed on to other processes

• Renewable via service APIs (RPC, HTTP)

• Revocable in server via service APIs

Page 19

read: O'Malley 2009, Hadoop Security Architecture

© Hortonworks Inc. 2015

HDFS IO: Block Tokens

Page 20

alice@REALM

Obtain ticket for namenode/nn@REALM

BlockToken

BlockToken

BlockToken: userId, (BlockPoolId, BlockId), keyId, expiryDate, access-modes

namenode/nn@REALM

open("file")

© Hortonworks Inc. 2015

service/host@REALM

Delegation Tokens delegate access

Page 21

alice@REALM

BlockToken

HDFS Delegation Token

BlockToken

HDFS Delegation Token

HDFS Delegation Token

namenode/nn@REALM

Token

Obtain ticket for namenode/nn@REALM

Request delegation token

© Hortonworks Inc. 2015

Launch Context

YARN app launch

Page 22

alice@REALM

HDFS Delegation Token

HDFS

resourcemanager/rm@REALM

nodemanager/_HOST@REALMalice

Launch Context

AM/RM

HDFS AM/RM

HDFS

HDFSHDFS

AM/RM

namenode/nn@REALM

Obtain ticket for resourcemanager/rm@REALM

Request delegation token

AM/RM

Token

Obtain tickvet for namenode/nn@REALM

AM/RM'

AM/RM'

AM/RM'

Refresh AM/RM

© Hortonworks Inc

That which must not be named: UGI

if(!UserGroupInformation.isSecurityEnabled()) { stayInALifeOfNaiveInnocence();} else { sufferTheEnternalPainOfKerberos();}

UserGroupInformation.checkTGTAndReloginFromKeytab();

UserGroupInformation.getLoginUser() // principal logged in asUserGroupInformation.getCurrentUser() // principal acting as

Page 23

© Hortonworks Inc

UGI.doAs()

UserGroupInformation bob = UserGroupInformation.createProxyUser("bob", UserGroupInformation.getLoginUser());

FileSystem userFS = bob.doAs( new PrivilegedExceptionAction<FileSystem>() { public FileSystem run() throws Exception { return FileSystem.get(FileSystem.getDefaultUri(), conf); } });

Page 24

© Hortonworks Inc

Hadoop RPC

@KerberosInfo(serverPrincipal = "my.kerberos.principal")public interface MyRpc extends VersionedProtocol { … }

public class MyRpcPolicyProvider extends PolicyProvider { public Service[] getServices() { return new Service[] { new Service("my.protocol.acl", MyRpc.class) }; }}public class MyRpcSecurityInfo extends SecurityInfo { … }

META-INF/services/org.apache.hadoop.security.SecurityInfoorg.example.rpc.MyRpcSecurityInfo

Page 25

© Hortonworks Inc

IPC Server: get the current user identity

Messages.KillResponse killContainer(Messages.KillRequest request) {

UserGroupInformation callerUGI;

try { callerUGI = UserGroupInformation.getCurrentUser();} catch (IOException ie) { LOG.info("Error getting UGI ", ie); AuditLogger.logFailure("UNKNOWN", "Error getting UGI"); throw RPCUtil.getRemoteException(ie);}

…

Page 26

© Hortonworks Inc

IPC Server: Authorize

String user = callerUGI.getShortUserName();

if (!checkAccess(callerUGI, MODIFY)) { AuditLog.unauthorized(user, KILL_CONTAINER_REQUEST, "User doesn't have permissions to " + MODIFY); throw RPCUtil.getRemoteException( new AccessControlException( + user + " lacks access " + MODIFY_APP.name()));}

AuditLog.authorized(user, KILL_CONTAINER_REQUEST)

Page 27

© Hortonworks Inc. 2015

SASL: RFC4422

Page 28

© Hortonworks Inc.

REST: SPNEGO (+ Delegation tokens)

Page 29

• Jersey + java.net• httpclient? “if lucky it'll work”

HADOOP-11825: Move timeline client Jersey+Kerberos+UGI support into a public implementation

© Hortonworks Inc.

Testing

Page 30

© Hortonworks Inc.

Error messages to fear

Art: Andrés Álvarez Iglesias

Failure unspecified at GSS-API level (Checksum failed)

No valid credentials provided (Failed to find any Kerberos tgt)

Server not found in Kerberos database

Clock skew too great

Principal not found

No valid credentials provided (Illegal key size)

© Hortonworks Inc.

Topics Avoided Not Covered

• Zookeeper

• JAAS

• Trying to use HTTPS in a YARN application

• Trying to use Full REST in a YARN application

• System properties to debug Kerberos & SPNEGO

• Group management

• HADOOP_PROXY_USER

Page 32

© Hortonworks Inc.

gitbook.com/@steveloughran

Questions?

Art: Andrés Álvarez Iglesias

© Hortonworks Inc.

Zookeeper

• SASL to negotiate security:System.setProperty("zookeeper.sasl.client", "true");

• Permissions are not transitive down the tree

Page 34

List<ACL> perms = new ArrayList<>();if (UserGroupInformation.isSecurityEnabled()) { perms(new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.AUTH_IDS)); perms.add(new ACL(ZooDefs.Perms.READ,ZooDefs.Ids.ANYONE_ID_UNSAFE));} else { perms.add(new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.ANYONE_ID_UNSAFE));}zk.createPath(path, null, perms, CreateMode.PERSISTENT);

© Hortonworks Inc

System Properties for debugging

-Dsun.security.krb5.debug=true-Dsun.security.spnego.debug=true

export HADOOP_JAAS_DEBUG=true

Page 35

© Hortonworks Inc.

Services

• RPC authentication via annotations & metadata in JAR• YARN Web UIs: rely on RM proxy for authentication• Authentication != Authorization• Add audit logs on service endpoints• YARN services: come up with a token refresh strategy:keytab everywhere; keytab in AM; update from client

Page 36

© Hortonworks Inc.

JAAS

• Java Authentication and Authorization Service• Core Kerberos classes and types (Principal)• Text files to configure

–Different for different JVMs–Need to double escape \ for windows paths

• UGI handles setting up a JAAS context & logging in

Page 37

© Hortonworks Inc.

Glossary

• Simple Authentication and Security Layer (SASL)• GSSAPI Generic Security Service Application Program Interface (RFC-2743+ others)

• JAAS: Java Authentication and Authorization Service• Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO)

Page 38