Dev Ops in Big Data

Sujesh Chirackkalgithub: https://github.com/sujeshchirackkalLinkedIn: https://in.linkedin.com/in/sujesh-chirackkal-60270b51

Dev OPS Eco System

Continuous Improvemen

t

Continuous Delivery

Process Simplification

Agile

Continuous Integration

“It is painful to upgrade, let us not do it”

Are we ready to upgrade our cluster and applications any time

“It is going to impact business, let us live with current version for some more time”

Challenges• Environment readiness

• Changes without backward compatibility

• Time, Resource constraints etc.

Automated system tests using

• Create a pseudo cluster, as a Docker container, of the new version to be used.

• Write code to perform automated system/integration test along with unit test cases.

What we did

/** * testSettings to invoke as "it:run" */def testSettings(): Seq[Setting[_]] = { // override the run settings, run command on sbt will invoke the tests run := { val dockerImg = docker.dockerImage.value

val containerId = dockerImg.startContainer().substring(0, 12) if(dockerImg.checkIfContainerReady(containerId)) { val result = dockerImg.runJob(containerId) } else { System.exit(-1) } dockerImg.stopContainer(containerId) }}

Leveraging SBT(Scala Build Tool) test settings

class DockerImage(dockerImageName: String, dockerTag: String, mountDir: String, baseDir: File) { val localDir = (baseDir)

def startContainer(): String = { Process( "docker" :: "run" :: "-d" :: ”-P” :: "-v" :: s"${localDir}:${mountDir}" :: s"${dockerImageName}:${dockerTag}" :: Nil ) !! }

def runJob(containerId: String) = { Process ("docker" :: "exec" :: "-t" :: s"$containerId" :: "/bin/bash" :: "/var/demo/src/main/scripts/run.sh" :: Nil)! }

def stopContainer(containerId: String): Unit = { Process("docker" :: "stop" :: s"$containerId" :: Nil)! }

def checkIfContainerReady(containerId: String): Boolean = { val status = Process("docker" :: "exec" :: "-t" :: s"$containerId" :: "/bin/bash" :: "/var/demo/src/main/scripts/checkContainer.sh" :: Nil)!

if (status != 0) { stopContainer(containerId) false } else true }}

Managing Docker lifecycle

Linux package mapping plugins in SBT(Scala Build Tool)

• Final deployable is a single file (an rpm file)

• A single command to deploy the application (yum install or yum update)

• Application artifacts are immutable on the deployed server (only root can access), also you need root to deploy the same.

Linux package mapping plugins in SBTlinuxPackageMappings <<= (assembly, name, version, sourceDirectory, target) map { (fatJar: File, name: String, version: String, src: File, target: File) => Seq( packageMapping( fatJar -> s"/usr/share/myapp/${version}/${fatJar.getName}" ).withPerms("0755"))}

lazy val myRunScript = TaskKey[File] (”my-run-script","Run Script for the application")

myRunScript <<= (assembly, version, sourceDirectory, target) map { (fatJar: File, version: String, src: File, target: File) => createScript(target / ”my_run_script", IO.read(src / "main/scripts/my_run_script.sh") .replace("%myjar%", fatJar.getName))}

linuxPackageMappings <++= (myRunScript , version) map { (script: File, version: String) => Seq( packageMapping( script -> s”/usr/share/myapp/${version}/${script.getName}" ).withPerms("0755"))}

Questions ???

Thank you