End-to-End Encryption of Data-at-Rest - · Notes: Performance is in Internal Throughput Rate (ITR) ... DataStage* DB2* GDPS Global Business Services* IBM* IBM (logo)* InfoSphere

End-to-End Encryption of Data-at-Rest

for Linux on IBM Z and LinuxONE

Reinhard Buendgen [email protected]

Product Owner Security for Linux on Z

IBM Z / ZSP03160-USEN-38 / July 17, 2017 / © 2017 IBM Corporation

mailto:[email protected]

2 © 2017 IBM Corporation

Trademarks

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations

such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements

equivalent to the performance ratios stated here.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance

characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business

contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM

products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

This information provides only general descriptions of the types and portions of workloads that are eligible for execution on Specialty Engines (e.g, zIIPs, zAAPs, and IFLs) ("SEs"). IBM authorizes customers to use IBM SE only to execute the

processing of Eligible Workloads of specific Programs expressly authorized by IBM as specified in the “Authorized Use Table for IBM Machines” provided at www.ibm.com/systems/support/machine_warranties/machine_code/aut.html (“AUT”). No

other workload processing is authorized for execution on an SE. IBM offers SE at a lower price than General Processors/Central Processors because customers are authorized to use SEs only to process certain types and/or amounts of workloads as

specified by IBM in the AUT.

* Registered trademarks of IBM Corporation

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

IT Infrastructure Library is a Registered Trade Mark of AXELOS Limited.

ITIL is a Registered Trade Mark of AXELOS Limited.

Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its

subsidiaries in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.

UNIX is a registered trademark of The Open Group in the United States and other countries.

VMware, the VMware logo, VMware Cloud Foundation, VMware Cloud Foundation Service, VMware vCenter Server, and VMware vSphere are registered trademarks or trademarks of VMware, Inc. or its

subsidiaries in the United States and/or other jurisdictions.

Other product and service names might be trademarks of IBM or other companies.

CICS*Cognos*DataStage*DB2*GDPS

Global Business Services*IBM*IBM (logo)*InfoSphereMaximo*

MQ*Parallel Sysplex*QualityStageRational*Smarter Cities

XIV*zEnterprise*z/OS*z Systems*z/VM*

SPSS*System Storage*System x*Tivoli*WebSphere*

z/VSE*

http://www.ibm.com/systems/support/machine_warranties/machine_code/aut.html

Contents

3

1. What is end-to-end data-at-rest encryption?

2. Data-at-rest-encryption in Linux on Z

3. Linux Unified Key Setup

4. CPACF protected key crypto

5. dm-crypt with protected keys

6. Best practices for using dm-crypt


4

What is End-to-EndData-at-Rest Encryption?


5© 2017 IBM Corporation

Attack Points to Data

Questions

• Where is data decrypted/encrypted?

• Who generates keys?

• Who owns (can access) the keys?

• Backups? Data migration?

Storage server Server

VS

application

OS-kernel

HVSANcache

adapter adapter

Attack points

• storage server

• SAN

• Server / HV /VS

• insider / outsider


Data Encryption on Storage Subsystem

Questions

• Where is data decrypted/encrypted? storage server

• Who generates keys? storage/OS

• Who owns (can access) the keys? storage/OS admin


VS

application

OS-kernel

HVSANcache

adapter adapter

Attack points

• storage server -- (secured)

• SAN – not secure

• Server / HV – not secure

• VS insider / outsider – not secure


application

SAN Encryption

Questions

• Where is data decrypted/encrypted? adapter

• Who generates keys? system admin

• Who owns (can access) the keys? system admin


VS

OS-kernel

HVSANcache

adapter adapter

Attack points

• storage server -- not secured

• SAN -- secured

• Server / HV -- not secured

• VS insider / outsider – not secured


application

NAS Network Encryption

Questions

• Where is data decrypted/encrypted? OS-Kernel & NAS

• Who generates keys? OS admin

• Who owns (can access) the keys? OS admin, NAS

NAS Server

VS

OS-kernel

HV

Intranet

/Internetcache

adapter adapter

Attack points

• NAS -- not secured

• Network -- secured

• Server / HV – (secured)

• VS insider / outsider – not secured


End-to-end Data Encryption

Questions

• Where is data decrypted/encrypted? applikation or kernel

• Who generates keys? app or OS admin

• Who owns (can access) the keys? app or OS admin


VS

application

OS-kernel

HVSANcache

adapter adapter

Attack points

• storage server -- secured

• SAN -- secured

• Server / HV – (secured)

• VS insider / outsider -- not secured


End-to-end Data Encryption from an SSC

Questions

• Where is data decrypted/encrypted? application or kernel

• Who generates keys? app or OS admin

• Who owns (can access) the keys? app or OS admin


SSC

application

OS-kernel

HV (= secure LPAR)SANcache

adapter adapter

Attack points

• storage server -- secured

• SAN -- secured

• Server / HV – secured

• VS insider / outsider -- hardened

11

Data-at-Rest Encryption in Linux on Z



Linux (on Z) Encryption of Data at Rest

• dm-crypt: block device / full volume encryption

• uses kernel crypto

• granularity: disk partition / logical volume

• ext4 with encryption option: file system encryption


• granularity: file, directory, symbolic link

• Spectrum Scale (GPFS) with encryption option: file encryption

• uses GSKit or Clic crypto libraries

• granularity: file

• DB2 native encryption: data base encryption

• uses GSKit crypto library

• NFS v4 with encrytion option: encryption of file transport


• SMB v3.1: encryption of file transport


kernel crypto automatically

uses CPACF for AES if the

module aes_s390 is loaded

GSKit and latest versions of

Clic use CPACF for AES

e2

e e

ncry

ptio

nn

etw

ork

encry

ption

13

SAN

vdisk disk

FSdm-crypt

dm-crypt Overview dm-crypt

– a mechanism for end-to-end data encryption

– data only appears in the clear when in program

kernel component that transparently

– for a whole block device (partition or LV)

• encrypts all data written to disk

• decrypts all data read from disk

How it works:

– on opening a volume

• dm crypt is told which key and cipher to use

– dm-crypt module uses in kernel-crypto

• can use z System HW if aes_s390 module loaded

– AES-CBC

– XTS-AES (recommended)

application

Linux kernel

file system

block device driver

Linux

on z14

XTS-AES

is vey fast

14

Linux File System Stack

application

kernel

disk disk

physical

block DD

physical

block DD

logical block DD

logical block DD

.

.

.

page

cache

file system (e.g. ext4)

virtual file system

I/O system call

(open, read,

write)

standard I/O

(through page cache)

direct I/O

(bypassing

page cache)

direct I/O to

device

(e.g. swap)

layers of logical

device drivers:

logical volumes,

RAID, multipath

15

Linux File System Stack with dm-crypt

application

kernel

disk disk

physical

block DD

physical

block DD

logical block DD

logical block DD

page

cache

file system (e.g. ext4)

virtual file system

I/O system call

(open, read,

write)

standard I/O

(through page cache)

direct I/O

(bypassing

page cache)

direct I/O to

device

(e.g. swap)

layers of logical

device drivers:

logical volumes,

RAID, multipath

+ dm-crypt

dm-crypt

en

cry

pte

dcle

ar

text

What Volumes can be Encrypted by dm-crypt?

YES: block devices

– Disks:

• SCSI disks

– Partitions of

• ECKD DASDs

• SCSI disks

– muti-path devices

– (LVM2) logical volumes

– loop back devices

– …

NO:

– full ECKD DASDs

– network file systems like NFS

• you can create a loopback device based on

a file in a network file system


How to Set-up a dm-crypt Plain Format Volume

1: generate key and store it in safe location

– dd if=/dev/random of=keyfile …

2: open dm-crypt volume and assign it a virtual volume name

– cryptsetup open –type plain …

• raw volume, virtual volume name, keyfile, key length, cipher

– creates virtual volume in /dev/mapper

3: use virtual volume

– mkfs (or mkswap)

– mount (or swapon)

– any kind of standard I/O

– do not use raw volmue directly

/dev/dasdb1

/dev/mapper/

sec_dev

for each virtual

dm-crypt

volume, the

kernel

maintains raw

volume and

encryption info

once

with

every

boot

business

as usual

/mount_point

associate

key with

volume!

18

Linux Unified Key Setup


The LUKS Format

19

Goal:

– provide a self containd volume format that

• associates a volume contents with

• an encryption key

• a cipher

• LUKS2: a dm-crypt sector size

– Protect encryption key

• with a password

– Goody:

• allow sharing of encryption key by up to 8 password holders

– Provide tooling to

• format volume

• open volume accrding to its format


The result of dm-crypt data enyption

depends on the sector size


On CPACF Performance

CPACF performance depends on the size of buffers being en-/decrypted

the larger the buffer the better the performance

– best performance with ≥ 4kB buffers

dm-crypt always used 512 byte buffers

starting with LUKS2 (kernel 4.12 and cryptsetup 2.0):

– dm-crypt can use 4kB buffers (= sectors)

– both for LUKS2 and plain format

512b 4kB

z13 1 1.5

z14 4 13

relative AES GCM/XTS in

memory performance


How to Set-up a dm-crypt LUKS Volume 1: format raw volume as dm-crypt volume

– cryptsetup luksFormat …

• cipher, key length, hash, pass phrase

– writes dm-crypt superblock to disk

2: open dm-crypt volume and assign it a virtual volume name

– cryptsetup luksOpen …

• dm-crypt volume, virtual volume, pass phrase

– creates virtual volume in /dev/mapper


– mkfs (or mkswap)

– mount (or swapon)

– any kind of standard I/O

– do not use raw volmue directly

/dev/dasdb1

/dev/mapper/

sec_dev

1

2

kernel translates I/O

to virtual volume into

I/O to raw volume

and performs

decryption/encryption

kernel stores

raw and

encryption

info for virtual

volume

once

with every

boot

business

as usual

superblock

contains

wrapped key

22

Example setting up a dm-crypt LUKS volume format a partition as encrypted volume -- only required once

# cryptsetup luksFormat -c aes-xts-plain64 -s 512 /dev/dasdb1

– password to encrypt (random) key must be supplied

open encrypted volume as a device mapper volume# cryptsetup luksOpen /dev/dasdb1 sec_dev

– password must be entered

– new virtual volume /dev/mapper/sec_dev will be created

create file system -- only required once# mkfs.ext4 /dev/mapper/sec_dev

mount virtual device mapper volume# mkdir /home/mySecDir

# mount /dev/mapper/sec_dev /home/mySecDir

use it, e.g.:# echo ’what is secret’ > /home/mySecDir/mysecret

# ls /home/mySecDir

# cat /home/mySecDir/mysecret

luksOpen

mount

24

The LUKS(2) Format LUKS = Linux Unified Key Setup

Describes format of disk superblock = initial sector(s):

– magic, version

– cipher, mode of operation, key length, IV generation

• used to wrap (= encrypt) dm-crypt master keys

• used for data encryption

– hash, salt, iteration

• used when generating key from password (via PBKDF2)

• used to compute dm-crypt master key digest

– dm-crypt master key digest

– uuid

– 8 key descriptions

– 8 sections containing wrapped enflated keys

.

Allows for up to 8

passwords to

wrap the dm-

crypt master key

…

…

LUKS2

supports for

4K dm-crypt

sectors

25

CPACF Protected Key Crypto



Clear Key vs. Secure Key Cryptography

master key

hardware security module

(HSM) -- tamper proof

E.g. Crypto Express Adapter

do not confuse

secure und secret keys

secure key cryptoclear key crypto

secure key

clear key =

plain text key


CPACF Protected Keys IBM Z function of the CPU (CPACF)

each virtual server (LPAR or guest) has a hidden master key

– not accessible from operating system in LPAR or guest

protected key: a key wrapped by the hidden LPAR/guest master

protected key tokens can be generated

– from secure keys using CEX Adapter (secure)

IBM Z CPU can compute symmetric encryption for protected keys

– pro

• no clear keys in operating system memory

• fast, encryption/decryption at CPU/CPACF speed, no I/O needed

– cons

• „hiding“ of master key not as good as in HSM (not tamper proof)

• not certified as HSM

• the LPAR/guest master key is not persistent

– need to store (secure) key to derive protected key from

master key

CPU/FW

memory not accessible by OS

protected key

28

pkey kernel modules

– uses zcrypt device driver (ap)

– generates secure keys

– transfroms secure keys into protected keys

cipher “paes” implements protected key AES

– paes_s390 kernel module

– takes secure key as argument

– generates and caches equivalent protected key

on opening the dm-crypt volume with paes cipher:

– transform secure key into equivalent protected key

– protected key encryption/decryption at CPACF speed

Kernel Support for Protected Keys

29

dm-crypt with Protected Keys



SAN

vdisk disk

FSdm-crypt

End-to-End Data at Rest Encryption with Protected Key dm-crypt

E2E data encryption

– The complete I/O path outside the kernel is encrypted:

• HV, adapters, links, switches, disks

dm-crypt

– a mechanism for end-to-end data encryption

– data only appears in the clear in application

Linux kernel component that transparently

• for all applications

• for a whole block device (partition or LV)

– encrypts all data written to disk

– decrypts all data read from disk

How it works:

– uses in kernel-crypto

• can use IBM Z CPACF Protected KeyCrypto:

– XTS-PAES

– encrypted volumes must be opened before usage

• opening provides encryption key to kernel• establishes virtual volume in /dev/mapper

application

Linux kernel

file system

block device driver

Linux

PKEYPAES

http://public.dhe.ibm.com/software/dw/linux390/docu/l5n1dc00_a_quick_start.pdf


Using the PAES with dm-crypt – Plain Format

generate a file with a secure key# zkey generate seckey.bin –xts

– requires access to CEX[5|6]C adapter

open block device as device mapper volume# cryptsetup open --type plain --key-file seckey.bin \\

--key-size 1024 --cipher paes-xts-plain64/dev/dasdb1 \\

plain_enc

– new virtual device mapper volume /dev/mapper/plain_enc will be created

– requires access to CEX[5|6]C adapter

use new device mapper volume

– (only once) create file system: # mkfs.ext4 /dev/mapper/plain_enc

– mount: # mount /dev/mapper/plain_enc /mount_point

– access: # echo “hello world” > /mount_point/myfile

/dev/dasb1

/dev/mapper/

plain_enc

/mount_point

once

with

every

boot

BAU

seckey.bin


Using the PAES with dm-crypt – LUKS Format0: insert new kernel modules during boot

requires access to CEX5C or CEX6C adapter

1: format raw volume as dm-crypt volume

cryptsetup luksFormat …

– cipher, key length, hash, password

– use “paes” instead of “aes”

writes dm-crypt header to disk

2: open dm-crypt volume and assign it a virtual

volume name

cryptsetup luksOpen …

– dm-crypt volume, virtual volume, passphrase

(needs not be protected)

creates virtual volume in /dev/mapper


mkfs (or mkswap)

mount (or swapon)

any kind of standard I/O

do not use raw volmue directly

/dev/dasdb1

/dev/mapper/

sec_dev

1

2

kernel translates I/O

to virtual volume into

I/O to raw volume

and performs

decryption/encryption

kernel stores

raw and

encryption

info for virtual

volume

once

with

every

boot

business

as usual

header contains

wrapped secure

key

once

modification

of cryptsetup

submitted

upstream but

not yet

accepted


LUKS/dm-crypt with Protected Keys - Components

35

Best Practices for Using dm-crypt



Best Practices with (Protected Key) dm-crypt

use /etc/cryptsetup

– to configure automated opening of volumes

use dm-crypt volumes as LVM physical volumes

– allows transparent data migration

protection against data loss

– to deal with LUKS superblock corruption

• back-up dm-crypt superblock

– to deal with HSM loss

• make sure you have a back-up adapter with common master key

• or

- generate secure key from clear key in clean room environment

- and store clear key in safe

37

/etc/crypttab

configuration file to describe how dm-crypt volumes are to be opened

will be read and interpreted before /etc/fstab

format

– each line decribes a dm-crypt volume:

• <dm-crypt volume> <path of block-device> <password file>|none [options]

• options may be set to describe volatile swap and tmp volumes

– example lines

sec_dev /dev/dasdb1 /root/PWs/sec_dev.pw

sec_swap /dev/dasdb2 none cipher=aes-xts-plain64,size=512,hash=sha512,swap

plain_vol /dev/dasdd1 /root/seckey.bin plain,cipher=paes-xts-plain64,\\

hash=plain,size=1024

38

Good/Best Practice: Logical Volumes Use logical volumes:

– allows multi-pathing,

– striping, SW-RAID

– dynamic growth of volumes

How to build logical volume?

– turn block device into “LVM physical volume”

– # pvcreate <device>

– create a volume group

– # vgcreate <volume group> <physical volume>

– add further physical volumes to volume group as needed

– # vgextend <volume group> <physical volume>

– create logical volume

– # lvcreate -L <size> <volume group> <logical volume>

– logical volume can now be accessed as

/dev/<volume group>/<logical volume>

VG

LV1 LV2

PV1 PV2 PV3

V1 V2 V3

39

Using dm-crypt Volumes as Physical VolumesUse virtual dm-crypt devices

as input to vgcreateMigrate data from unencrypted volume to dm-crypt volume

– 1: add dm-crypt based physical volume to volume group:

# vgextend VG DMV

– 2: migrate data from V1 to DMV:

# pvmove V1 DMVVG

LV1 LV2

PV1 PV2 PV3

DM

V1

DM

V2

DM

V3

V1 V2 V3

VG

LV1 LV2

PV1 PV2

DMV

V1 V2

1

2

works transparently

while applications

access LVs

40

Back-ups of Encrypted Data Back-ups at file system level

– e.g. with tar

– back-up is plain text encryption depends on

how back up is stored

Raw disk image

– e.g. from /dev/dasdb1

– back up both LUKS header & production data

– back-up is encrypted exactly as original disk

– self-contained as it contains LUKS header

Do not forget:

– plain format: always back-up key file

– LUKS: back-up of the LUKS header

luksOpen

mount

tar

dd

cryptsetup luksDump

encrypt

cp/tar


Questions?


Backup


LUKS/dm-crypt with Protected Keys - ComponentsComponents

new pkey kernel module for protected key

management

– generate secure key

– tansform secure key into protected key

new paes kernel module to perform protected key

encryption-decryption

– introduces paes cipher

dm-crypt kernel module (unchanged)

extended dm-crypt management tool cryptsetup

– recognizes paes cipher

– stores secure key into LUKS header

new zkey tool to

generate and manage secure keys

dm-crypt paes pkey

cryptsetup

CEX5CCPU

“paes”

Linux kernel

new new

modified

zkey


LUKS/dm-crypt Protected Keys - Processing

1. format volume

– write secure key and paes cipher in volume

header

2. open volume

– password not security relevant

– fetch secure key from volume header

– pass secure key to dm-crypt

– let paes transfrom secure key into protected

key

3. read/write data

– dm-crypt uses paes with protected key to

decrypt/encypt data

dm-crypt paes pkey

cryptsetup

CEX5CCPU

b) store sec key

into LUKS

header

a) generate secure key

“paes”

Linux kernel

zkey



1. format volume


header

2. open volume





key

3. read/write data


decrypt/encypt data

dm-crypt paes pkey

cryptsetup

CEX5CCPU

a) fetch

sec key

from LUKS

header

b) provide

sec key to dm-crypt

c) transform sec key

into prot key

“paes”

Linux kernel



1. format volume


header

2. open volume





key

3. read/write data


decrypt/encypt data

dm-crypt paes pkey

cryptsetup

CEX5CCPU

read/write:

en/decrypt

data“paes”

Linux kernel

Documents

End-to-End Encryption of Data-at-Rest - · Notes: Performance is in Internal Throughput Rate (ITR) ... DataStage* DB2* GDPS Global Business Services* IBM* IBM (logo)* InfoSphere