Coscup

Better node.js software

release practice using open

source tool

????????

Software release to production is such a hard practice, especially considering the business

impact and the quality of software . I will demonstrate how to use open source tools to

enhance a node.js web software release process and ensure the better release quality.

The open source tools I will mention will includes Jenkins, Git, Ubuntu dpkg system, node.js

language , ruby , rake, jshint , nodeunit, expresso , YETI, Selenium, and mongodb

?, ??????? 100+ ???? = =

Web Software release best

practice

Agenda

Who am I ?

Release engineering in Y! Shopping

Process

Testing

Deploy Tools

Configuration Management

Who am I ?

thegiive

• A open source lover

• Release manager of Taiwan Yahoo! Shopping site

• Continuous study on software development, release engineering and software quality

• Fall in love with Ruby, VIM, Puppet and Rake

• Trying to do more work rather than write more blog

Release engineering in Y! Shopping

Common image about release

engineer

Yahoo! Shopping release manager

integration hub for more complex software development teams, sitting at the cross between

development, product management, quality assurance and other engineering efforts, also known

as DevOps

DevOps

Release???1 : ???

??????????????

• RD Team ???? , PHP Team , .Net Team , Search Team

• User Team ???? , ?? Team , ?? Team , Search team

Release???1 : ???

?????

Release???2 : bug????

From Book "Beautiful Testing" Chaptor 2

TestROI : ????????

• ACPB : Average Cost of a Production Bug

• ACTB : Average Cost of a Test Bug

• TBN : Total Bug Number

• TDBC : Total Defect Bug Cost

Taiwan Shopping ? ACPB?????? Test ROI ???

Test ROI ??????????? testing ????

Release???2 : bug????

Release???3 : User????

• ??RD?bug?QA????????SE??????RM?????

• User ???? monitor script , ????????????

???? Release Manager , not release engineer

Release???4 : ??????

PHP Team

• Dev process : Scrum

• Source Code Management : Subversion

• Language : PHP and partial C++

• OS : Linux

.Net Team

• Dev process : Waterfall

• Source Code Management : VSS

• Language : ASP.Net and ASP

• OS : Windows

????????????? ... ?????? 100??

release ?release 2000 ??

item

??????

Release quality ??

Y axies stands for ehnancement number / release number

Throughput ??

Y axies stands for

release item / release number

User ?? , ???????

??Release Quality ??????

• 2011 Q4 24? release ??, ? 12??(02:00 ~ 06:00) release

• ???? user ?? manager ?????

??????????????? , ??????? release ???????, ????? 6???

?? release qulaity ???user ???????????

• 2012 Q2 11? release ??, ? 0??(02:00 ~ 06:00) release

• 2012 Q2? 1??07:30 release

????????

Process

Communication : No Surprise !

• ???????????, ?????????????????????? briefing and retrospective , ??????? 18%

• Communication ???? 9???

???????

?????????

• ????

• ???

• double confirm ??????????

??

• ????

• ?? 30min ??, 30???????????? 20000 NTD

??????

• ?????? sharing information ?????

????????

?????????

• Information syncing : 40%

• Discussion : 30%

• decision making : 30%

?????

• Information sync : 5% => ?????????

• Discussion : 15%

• decision making : 10%

Convention over configuration

Release??

• Release ????? ???????????

• ??????? release ??

• ????????? release

• Release ???????? release briefing

• Release ??? Code Freeze

• ???? cost , ?????? retrospective

??????????? sync ??

• ???? Release Plan

• ????????? Plan ??

◦ Item ????,???? , ????

◦ Item ?????? , ???? , ????

◦ ??????? item ????????????

release briefing ?????

• Information sync : 5% => information double confirm

• Discussion : 15% => ???? sync , ??????????

• decision making : 10% => ??sync??????????????

???????

• ???????? ??? release????? ????email

• ????????????????? , ????????????

• ????? , ???? surprise , ???????

Testing

Release Manager ???

• Unit Test : nodeunit , YETI

• Automation : Selenium

• CI ( Continuous Integration ) : Jenkins

????????????????

Unit Test ?????

?? code ???? , 100% RD ???

Automation

• ??????? SLA (Service-level agreement)

• ???? ?? automation coverage rate ??? SLA ? 100%

• ?? release ?????? automation

• Presentation layer ? coverage rate ???? 100% , ???????

Continuous Integration

• ??????? Unit Test and Automation , ???????

• CI ????????????? Unit Test , Automation

• ?????? CI ? auto build ??

Deploy Tool

???SE/Ops ??? Deploy Tool

??: ???? Yinst/Igor

Use os level package system

• Debian/Ubuntu : dpkg

• RedHat/Centos/Fedora : rpm

• FreeBSD : port

• Windows : MSI

Don't use language level package system , ????????? relay on ???????? server

• Webservr : apache, nginx

• Cache : memcached , redis

• SQL : MySQL , PosgreSQL , MongoDB

????? Deploy Script

?????????? script ? deploy code

1 apt-get install npm

2 apt-get install pip

3 apt-get install rubygem

4 apt-get install nginx #web server

5 apt-get install memcached #cache server

6 npm install -g express # Web framework

7 npm install -d

8 pip install pykml

9 gem install amqp # node.js has no rabbitmq client , so use ruby XD

10 apt-get install rabbitmq-server # rabbitmq server

11 wget http://mydev.server/mycode_package.tgz # get my code from dev server

12 tar zxcf mycode_package.tgz # unpack the code package

• ???? hybrid ?? package system , package dependency ?????

• SE/Ops team ?? language package system ??? , ???? RD ????

?? Deploy Script

????? package system , ?????? package version

1 apt-get install nginx=0.1.19-1

2 apt-get install memcached=1.4.13-0ubuntu2

3 apt-get install node-express=2.5.9-1

4 apt-get install pykml=0.0.3

5 apt-get install ruby-amqp=0.9.5-1.

6 apt-get install rabbitmq-server=2.8.4-1

7 apt-get install mycode_package=1.2.3 # ???? code ?? package

??

• SE/Ops team ??????? deploy ???

• CI server ????

???? pkg version

Production System ? ???? pkg system ? depentency

How to pack your package

• ?? code ????? (Code,Config,template)

• ??? code ??? prodcut ????? (??,??????,???,????)

• ????????? (??lib ,??lib ,?? config , ??config )

• ???????????? ( 100? , 10? , 1? , ???? , ???? )

???? Framework ???

Change Management

??? code ???????

• mycode-1.2.3.zip : ?????? change ???? ???? readme

PKG list : ?? SE/Ops ?????

• Upgrade : functionA-config-1.2.3-2012.07.08

• Upgrade : Member-template-2.3.2-2012.07.11

• Donwgrade : functionC-code-3.2.1-2012.03.05

• Upgrade : ShoppingCart-lib-100W-1.2.3-2012.07.20

• Upgrade : WS-stock-lib-200W-3.2.1-2012.07.21

Package pack script ???

Script best practice

• ???

• ? package ???? source code version ???

• Version ??????

• Package ????? package server

• Readme ???????? svn log

Package Version 2.3.1-2012.08.12

[thegiive v123] add some slides change

[RD2 v121] change function a

Configuration Management

debconf

???? config package

• mypkg-config.2.2.3-2012.07.17

• ?????? config ? ????? readme

?????? config ??

1 echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections

2 echo "mypkg mypkg/ws_host string ws1.coscup.org"|debconf-set-selections

3 echo "mypkg mypkg/title string coscup"|debconf-set-selections

4 debian_frontend=noninteractive dpkg-reconfigure

package ??? postinst

1 db_get mypkg/use_memached

2 echo $RET #??? debconf ?????

3 do_config_change($RET)

??????SE/Ops?????

Cache Server ???SE?????? disable cache

1 echo "mypkg mypkg/use_memached boolean false"|debconf-set-selections

2 debian_frontend=noninteractive dpkg-reconfigure

WS1.coscup.org ??????SE???? WS2.coscup.org

1 echo "mypkg mypkg/ws_host string ws2.coscup.org"|debconf-set-selections

2 debian_frontend=noninteractive dpkg-reconfigure

???? title , SE ????? XD

1 echo "mypkg mypkg/title string coscup.org"|debconf-set-selections

2 debian_frontend=noninteractive dpkg-reconfigure

• ????RD ???SE?Daily work?????? RD , SE ??????

• ?? incident ??? , SE ????config ???? , RD??? SE?????

?? mobile login server ??

• ?? mobile loging server ?????????????????

• ?????????? RD ???????RD,SE,??????

• ???? config change , ????? mobile login ?? Web?? login ??

Debconf ????...???

????????deploy script

1 # pkg

2 export debian_frontend=noninteractive

3 apt-get install -y nginx=0.1.19-1

4 apt-get install -y memcached=1.4.13-0ubuntu2

5 apt-get install -y node-express=2.5.9-1

6 apt-get install -y pykml=0.0.3

7 apt-get install -y ruby-amqp=0.9.5-1.

8 apt-get install -y rabbitmq-server=2.8.4-1

9 apt-get install -y mycode_package=1.2.3

10 # config

11 echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections

12 echo "mypkg mypkg/ws_host string ws1.coscup.org"|debconf-set-selections

13 echo "mypkg mypkg/title string coscup"|debconf-set-selections

14 dpkg-reconfigure

• ??????? script ? deploy ?????????? , VM , Cloud instance

• script ???? non-interactive mode ? disable ??????? step

• ?? CI server ?? auto build , auto config

deploy script ?? version

control

?? tag ??? deploy , rollback

7??1? release ? tag ? v2012_07_01 , 7??2? release ? tag ? v2012_07_02

1 # ???1? release

2 git tag v2012_07_01

3 exec_deploy_script

4 # ???2? release

5 git tag v2012_07_02

6 exec_deploy_script

? v2012_07_02 release ??????? rollback ??????(v2012_07_01)

1 git checkout v2012_07_01

2 exec_deploy_script

?? git diff ?????? change

????7??2? release ?????

1 git diff v2012_07_01 v2012_07_02

2 #git diff output

3 .....

4 -apt-get install -y mycode_package=1.2.3

5 +apt-get install -y mycode_package=1.3.1

6 -echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections

7 +echo "mypkg mypkg/use_memached boolean false"|debconf-set-selections

8 .....

9 -echo "mypkg mypkg/title string coscup"|debconf-set-selections

10 +echo "mypkg mypkg/title string coscup.org"|debconf-set-selections

11 .....

• package ????? , config management ????change ???

• ?? RD ??????? diff ???????????

?? git branch ?????

• ?? git ??? branch ???????? deploy script

• ???????? alpha , beta , staging , production ?? branch

??? alpha ???????? beta ??? git merge

1 git checkout beta

2 git merge alpha

3 ...solve confict...

4 git commit

One More Thing

Deploy System : Puppet

• Open source deploy system written by Ruby

• Client Server ????????? pure client

• Config is language, built by Ruby DSL !!!

• ??? template system ???? perfect configuration manage System

• ???OS ? package system , ???????? OS ????

• From Wikipedia ????: Google, Zynga, Sun, Wikipedia, Dell, Twitter...

Puppet Node config

In module

1 class ntpd {

2 package { "ntp": ensure => installed, }

3 file { "/etc/ntp.conf": owner => root, group => root,

4 require => package["ntp"] }

5 @service { "ntpd": ensure => running, enable => true,

6 subscribe => File["/etc/ntp.conf"] }

7 }

8 class ci{ package { "jenkins" : ensure => installed,} }

9 class Webservr{ package { "nginx" : ensure => installed,} }

In main config

1 node basenode { include ntpd }

2 node 'ci.coscup.org' inherits basenode {

3 include ci

4 }

5 node 'web.example.com' inherits basenode {

6 include webserver

7 }

Puppet Package config

?? Package ??

1 package {'vim': ensure => 'installed' }

2 package { "apache2": ensure => '2.2.22-1ubuntu1' }

3 package { 'mysql': ensure => installed,

4 provider => 'msi',

5 source => 'N:/packages/mysql-xxx.msi',

6 install_options => { 'INSTALLDIR' => 'C:\mysql-5.5' }, }

??????? Deploy Script

• Debian / Ubuntu : Apt-get

• RedHat / Fedora / CentOS : yum

• FreeBSD : port

• MacOS : macport

• Windows : msi

• Gentoo : portage

Puppet Template System

Configure Management ??? : ERB

??????? config file

1 $server_name = 'domain.coscup.org'

2 $server_port = 80

3 file {"/usr/conf/nginx.conf": mode => "664",

4 content => template("nginx/config.erb"), }

template

1 server {

2 listen <%= $server_port%>;

3 server_name <%= $server_name %>;

4 .....

5 }

Pure ruby config menas

1 Given I run "ssh localhost -p 2222" interactively

2 And I type "apache2ctl configtest"

3 And the exit status should be 0

• We can write unit test ... so we can TDD

• We can write rspec and cucumber ... so we can BDD

• We can commit our config in Hudson... so we can CI our config

??? Deploy Script ??

1 class web{ package { "nginx": ensure => '0.1.19-1' } }

2 class cache{ package { "memcached" : ensure => '1.4.13-0ubuntu2' }}

3 class kml_lib{ package { "pykml" : ensure =>'0.0.3' }}

4 class message_quene{

5 package { "ruby-amqp" : ensure => '0.9.5-1' }

6 package { "rabbitmq-server": ensure => '2.8.4-1' }

7 }

8 class code_package{

9 include kml_lib

10 include message_quene

11 package { "node-express" : ensure => '2.5.9-1' }

12 package { "mycode" : ensure => '1.2.3' }

13 }

14 node "w1.coscup.com" {

15 include web

16 include cache

17 include code_package

18 }

Q & A

Puppet Config

?? Command

1 exec { 'remove file' : command => 'rm /tmp/pup/etd' , path => '/bin' }

?? Windows Command

1 exec { 'wincommand': command => 'C:\Windows\..\powershell.exe -executionpolicy remotesigned -file C:\test.ps1', }

?? file

1 file{'/tmp/pup': ensure => 'directory'}

2 file{ '/tmp/pup/et': ensure => 'present', content => 'look at me',

3 require => File['/tmp/pup']}

4 file { "/etc/ntp.conf": owner => root, group => root,

5 mode => 0644, source => "puppet:///ntpd/ntp.conf",

6 require => package["ntp"]

7 }