Upgrading to Ruby 2.x

Joe Rafaniello

@jrafanie

2007

13

Agenda

1. History

2. Why upgrade?

3. Ruby 2.1

4. Ruby 2.0

5. "Fall cleanup" of old code

6. Slow tests

7. Building 2.0 appliances

8. Developer setup

9. Links

10. Questions?

History

Tue Nov 1 2011:

First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ

History

Tue Nov 1 2011:

First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ

Tue Apr 23 2013:

Ruby 1.9.3 finally...

History

Tue Nov 1 2011:

First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ

Tue Apr 23 2013:

Ruby 1.9.3 finally...

540 days???

History

Tue Nov 1 2011:

First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ

Tue Apr 23 2013:

Ruby 1.9.3 finally...

540 days???

Lesson learned: Don't wait to upgrade!

Why upgrade?

Why upgrade?

We're behind!!!

Ruby 1.9.3 is ending

In maintenance until February 23, 2014

Security only mode until February 23, 2015

Ruby 2.0.0 is nearly 20 months old

Ruby 2.1.0 is nearly 10 months old

Ruby 2.2.0 is scheduled for a Christmas release

Why upgrade? ... Because ruby 2.1!

Generational mark and sweep garbage collector

http://tmm1.net/ruby21-rgengc/

String#freeze - reuse String objects

Less objects == less memory == less GC time

Object allocation tracing

http://tmm1.net/ruby21-objspace/

https://github.com/srawlins/allocation_stats

Required keyword arguments

def returns method name

Exception#cause - ActiveRecord::StatementInvalid#cause -> Real error

More...

Why upgrade? ... Because ruby 2.1!

Useless benchmark

bundle exec rspec spec/models/ems_refresh/refreshers

1.9.3-p545

70.85s user 2.23s system 96% cpu 1:15.98 total71.21s user 2.22s system 96% cpu 1:16.27 total

Why upgrade? ... Because ruby 2.1!

Useless benchmark

bundle exec rspec spec/models/ems_refresh/refreshers

1.9.3-p545

70.85s user 2.23s system 96% cpu 1:15.98 total71.21s user 2.22s system 96% cpu 1:16.27 total

2.0.0-p576

54.02s user 2.03s system 95% cpu 58.980 total49.96s user 2.14s system 94% cpu 54.923 total

Why upgrade? ... Because ruby 2.1!

Useless benchmark

bundle exec rspec spec/models/ems_refresh/refreshers

1.9.3-p545

70.85s user 2.23s system 96% cpu 1:15.98 total71.21s user 2.22s system 96% cpu 1:16.27 total

2.0.0-p576

54.02s user 2.03s system 95% cpu 58.980 total49.96s user 2.14s system 94% cpu 54.923 total

2.1.3

36.52s user 2.79s system 91% cpu 42.930 total35.68s user 2.22s system 92% cpu 40.768 total

Why upgrade? ... Because ruby 2.1!

Example allocation information:

Line number

Number of allocations by object type

Such as:

Running: ./spec/controllers/application_controller/buttons_spec.rb:91

223730 Arrays @ .../activerecord/lib/active_record/result.rb:35

203280 Strings @ .../activerecord/lib/active_record/relation.rb:27

Why upgrade? ... Because ruby 2.1!

Example allocation information:

Line number

Number of allocations by object type

Such as:

Running: ./spec/controllers/application_controller/buttons_spec.rb:91

223730 Arrays @ .../activerecord/lib/active_record/result.rb:35

203280 Strings @ .../activerecord/lib/active_record/relation.rb:27

See Issue 241 and 762

But that's 2.1, let's get to 2.0 first...

Ruby 2.0 features

Faster Rails startup

Optimizations were made to speed up 'require'

Faster Rails startup

Optimizations were made to speed up 'require'

(master) (1.9.3-p545) + time bundle exec rake environmentbundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total

(master) (2.0.0-p576) + time bundle exec rake environmentbundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total

Faster Rails startup

Optimizations were made to speed up 'require'

(master) (1.9.3-p545) + time bundle exec rake environmentbundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total

(master) (2.0.0-p576) + time bundle exec rake environmentbundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total

25% faster loading of rails environment!

Faster Rails startup

Optimizations were made to speed up 'require'

(master) (1.9.3-p545) + time bundle exec rake environmentbundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total

(master) (2.0.0-p576) + time bundle exec rake environmentbundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total

25% faster loading of rails environment!

Most obvious when:

Running tests

Loading Rails console

Keyword arguments

Simplifies conventions:

Accessing option hash values

Default hash values

Optional / can't handle all cases

Keyword arguments

Example: EmsVmware#vm_connect_all

def vm_connect_all(vm, options={}) defaults = { :onStartup => false } options = defaults.merge(options) vm_connect_disconnect_all_connectable_devices( vm, true, options[:onStartup], options[:user_event] )end

Keyword arguments

Example: EmsVmware#vm_connect_all

def vm_connect_all(vm, options={}) defaults = { :onStartup => false } options = defaults.merge(options) vm_connect_disconnect_all_connectable_devices( vm, true, options[:onStartup], options[:user_event] )end

With keyword arguments:

def vm_connect_all(vm, user_event: nil, onStartup: false) vm_connect_disconnect_all_connectable_devices(vm, true, onStartup, user_event)end

Keyword arguments

Invoking methods not changed

Valid on 1.9.3 / 2.0.0:

vm_connect_all(:vm_object1, :onStartup => true, :user_event => "event1")

vm_connect_all(:vm_object2)

vm_connect_all(:vm_object3, :user_event => "event3")

vm_connect_all(:vm_object3, user_event: "event3")

Keyword arguments

Shortcomings and gotchas:

if - valid hash key / invalid keyword argument

Required keyword arguments added in 2.1

http://magazine.rubyist.net/?Ruby200SpecialEn-kwarg

http://robots.thoughtbot.com/ruby-2-keyword-arguments

http://chriszetter.com/blog/2012/11/02/keyword-arguments-in-ruby-2-dot-0/

Module#prepend

Problem: We want to debug a slow method.

Module#prepend

Problem: We want to debug a slow method.

Wrap the method so can time it!

class Parent def run puts "Parent" endend

class Sub < Parent def run puts "Sub" super end

def run_with puts "DebugIt" run_without end

alias_method :run_without, :run alias_method :run, :run_withend

Module#prepend

Using alias_method:

class Parent def run puts "Parent" endend

class Sub < Parent def run puts "Sub" super end

def run_with puts "DebugIt" run_without end

alias_method :run_without, :run alias_method :run, :run_withend

Sub is a class with the slow run method...

Module#prepend

Using alias_method:

class Parent def run puts "Parent" endend

class Sub < Parent def run puts "Sub" super end

def run_with puts "DebugIt" run_without end

alias_method :run_without, :run alias_method :run, :run_withend

Sub is a class with the slow run method...

irb(main):01:0> Sub.new.run

DebugItSubParent

Module#prepend

Using alias_method:

class Parent def run puts "Parent" endend

class Sub < Parent def run puts "Sub" super end

def run_with puts "DebugIt" run_without end

alias_method :run_without, :run alias_method :run, :run_withend

Sub is a class with the slow run method...

irb(main):01:0> Sub.new.run

DebugItSubParent

YAY! But that's really dirty!

(alias_method_chain)

Module#prepend

Using alias_method:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent include DebugIt

def run puts "Sub" super endend

Module#prepend

Using include:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent include DebugIt

def run puts "Sub" super endend

Sub is a class with the slow run method...

Module#prepend

Using include:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent include DebugIt

def run puts "Sub" super endend

Sub is a class with the slow run method...

irb(main):002:0> Sub.ancestors

=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]

Module#prepend

Using include:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent include DebugIt

def run puts "Sub" super endend

Sub is a class with the slow run method...

irb(main):002:0> Sub.ancestors

=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]

irb(main):01:0> Sub.new.run

SubDebugItParent

Module#prepend

Using include:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent include DebugIt

def run puts "Sub" super endend

Sub is a class with the slow run method...

irb(main):002:0> Sub.ancestors

=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]

irb(main):01:0> Sub.new.run

SubDebugItParent

UGH, Sub's method comes first!

Module#prepend

Using include:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent prepend DebugIt

def run puts "Sub" super endend

Module#prepend

Using prepend:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent prepend DebugIt

def run puts "Sub" super endend

irb(main):002:0> Sub.ancestors

=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]

Module#prepend

Using prepend:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent prepend DebugIt

def run puts "Sub" super endend

irb(main):002:0> Sub.ancestors

=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]

irb(main):01:0> Sub.new.run

DebugItSubParent

Module#prepend

Using prepend:

module DebugIt def run puts "DebugIt" super endend

class Parent def run puts "Parent" endend

class Sub < Parent prepend DebugIt

def run puts "Sub" super endend

irb(main):002:0> Sub.ancestors

=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]

irb(main):01:0> Sub.new.run

DebugItSubParent

Ship it!

... but don't forget to call super!

Module#prepend

Using prepend:

Array of symbols: %i and %I

Array of symbols: %i and %I

irb(main):001:0> %i{vmware redhat microsoft}=> [:vmware, :redhat, :microsoft]

Array of symbols: %i and %I

irb(main):001:0> %i{vmware redhat microsoft}=> [:vmware, :redhat, :microsoft]

%I allows interpolation:

Array of symbols: %i and %I

irb(main):001:0> %i{vmware redhat microsoft}=> [:vmware, :redhat, :microsoft]

%I allows interpolation:

irb(main):002:0> prefix = "vm_"=> "vm_"

irb(main):003:0> %I{#{prefix}vmware #{prefix}redhat #{prefix}microsoft}=> [:vm_vmware, :vm_redhat, :vm_microsoft]

Refinements

Goal: localize monkey patches

I'm not going to explain them because:

Ruby's open classes

Many gotchas...

See Charles Nutter (@headius/jruby guy) explanation:http://blog.headius.com/2012/11/refining-ruby.html

Enumerable#lazy

Enumerable methods evaluate left to right

With lazy, chains of enumerations are evaluated right to left

Enumerable#lazy

Enumerable methods evaluate left to right

With lazy, chains of enumerations are evaluated right to left

Ruby may "cheat":

May skip creating intermediate objects

Large collection operations may be optimized

Enumerable#lazy

Example benchmark

(0...1000).select(&:odd?).take(5).to_a

(0...1000).lazy.select(&:odd?).take(5).to_a

What is this doing?

Enumerable#lazy

Example benchmark

(0...1000).select(&:odd?).take(5).to_a

(0...1000).lazy.select(&:odd?).take(5).to_a

What is this doing?

First 5 odd numbers

=> [1, 3, 5, 7, 9]

Enumerable#lazy

require 'benchmark/ips'

Benchmark.ips do |x| x.report("normal") { (0...1000).select(&:odd?).take(5).to_a } x.report("lazy") { (0...1000).lazy.select(&:odd?).take(5).to_a }end

Enumerable#lazy

require 'benchmark/ips'

Benchmark.ips do |x| x.report("normal") { (0...1000).select(&:odd?).take(5).to_a } x.report("lazy") { (0...1000).lazy.select(&:odd?).take(5).to_a }end

Calculating ------------------------------------- normal 1539 i/100ms lazy 5778 i/100ms------------------------------------------------- normal 15123.2 (±2.5%) i/s - 76950 in 5.091373s lazy 59797.4 (±4.0%) i/s - 300456 in 5.033123s

See http://patshaughnessy.net/2013/4/3/ruby-2-0-works-hard-so-you-can-be-lazy

__dir__

__dir__ path of the script without the filename

# cat test.rbputs __dir__

# ruby test.rb/Users/joerafaniello/Code/test

__dir__

__dir__ path of the script without the filename

# cat test.rbputs __dir__

# ruby test.rb/Users/joerafaniello/Code/test

We have 984 instances of File.dirname(__FILE__)!

WAT...Why?

Ruby 2.0 breaking changes and deprecations

Note: We're green on travis, so we're getting close...

Objects don't respond_to? to protected methods

Ruby 1.9.3:

respond_to?(symbol) => public and protected methods

respond_to?(symbol, true) => all methods

Objects don't respond_to? to protected methods

Ruby 1.9.3:

respond_to?(symbol) => public and protected methods

respond_to?(symbol, true) => all methods

Ruby 2.0.0

respond_to?(symbol) => public methods only

respond_to?(symbol, true) => all methods

Objects don't respond_to? to protected methods

class Worker protected

def run endend

Objects don't respond_to? to protected methods

class Worker protected

def run endend

Worker.new.respond_to?(:run)1.9.3 => true2.0.0 => false

Objects don't respond_to? to protected methods

class Worker protected

def run endend

Worker.new.respond_to?(:run)1.9.3 => true2.0.0 => false

Pass true as second argument...

Worker.new.respond_to?(:run, true)1.9.3 => true2.0.0 => true

Objects don't respond_to? to protected methods

class Worker protected

def run endend

Worker.new.respond_to?(:run)1.9.3 => true2.0.0 => false

Pass true as second argument...

Worker.new.respond_to?(:run, true)1.9.3 => true2.0.0 => true

See Pull #685 - default_value_for gem

UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"

UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"

Ruby 1.9.3

US-ASCII is the default encoding of ruby scripts

UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"

Ruby 1.9.3

US-ASCII is the default encoding of ruby scripts

Binary string literals become ASCII-8BIT:

UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"

Ruby 1.9.3

US-ASCII is the default encoding of ruby scripts

Binary string literals become ASCII-8BIT:

irb(main):001:0> require './test'=> trueirb(main):002:0> FOO.encoding=> #<Encoding:ASCII-8BIT>

UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"

UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"

Ruby 2.0.0

UTF-8 is the default encoding

UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"

Ruby 2.0.0

UTF-8 is the default encoding

Binary string literals become UTF-8 even if invalid:

irb(main):001:0> require './test'=> trueirb(main):002:0> FOO.encoding=> #<Encoding:UTF-8>irb(main):003:0> FOO.valid_encoding?=> false

UTF-8 is the default character encoding of rubyscripts

So, what's the problem?

Binary strings are used in many places for vm "fleecing"

Invalid UTF-8 encoded strings != raw binary:

UTF-8 is the default character encoding of rubyscripts

So, what's the problem?

Binary strings are used in many places for vm "fleecing"

Invalid UTF-8 encoded strings != raw binary:

irb(main):003:0> FOO.valid_encoding?=> falseirb(main):004:0> FOO == "\222dL\256".force_encoding("ASCII-8BIT")=> false

UTF-8 is the default character encoding of rubyscripts

Solutions:

Force binary on individual Strings:

1.9.3/2.0.0 compatible

Painful on files with many binary string literals

UTF-8 is the default character encoding of rubyscripts

Solutions:

Force binary on individual Strings:

1.9.3/2.0.0 compatible

Painful on files with many binary string literals

# cat test.rbFOO = "\222dL\256".force_encoding("ASCII-8BIT")

UTF-8 is the default character encoding of rubyscripts

Solutions:

Force binary on individual Strings:

1.9.3/2.0.0 compatible

Painful on files with many binary string literals

# cat test.rbFOO = "\222dL\256".force_encoding("ASCII-8BIT")

irb(main):001:0> require './test'=> trueirb(main):002:0> FOO.encoding=> #<Encoding:ASCII-8BIT>irb(main):003:0> FOO.valid_encoding?=> trueirb(main):004:0> FOO == "\222dL\256".force_encoding("ASCII-8BIT")=> true

UTF-8 is the default character encoding of rubyscripts

Solutions:

Use String#b on individual strings

Not compatible with ruby 1.9.3

Copies the String in ASCII-8BIT encoding

Note: String#force_encoding("ASCII-8BIT") modifies the receiver

UTF-8 is the default character encoding of rubyscripts

Solutions:

Add #encoding magic comment at top

1.9.3/2.0.0 compatible

Good option when binary strings are expected

UTF-8 is the default character encoding of rubyscripts

Solutions:

Add #encoding magic comment at top

1.9.3/2.0.0 compatible

Good option when binary strings are expected

# cat test.rb# encoding: US-ASCIIFOO = "\222dL\256"

UTF-8 is the default character encoding of rubyscripts

Solutions:

Add #encoding magic comment at top

1.9.3/2.0.0 compatible

Good option when binary strings are expected

# cat test.rb# encoding: US-ASCIIFOO = "\222dL\256"

irb(main):001:0> require './test'=> trueirb(main):002:0> FOO.encoding=> #<Encoding:ASCII-8BIT>

Descriptors except 0, 1, 2 are closed in childprocesses

Prevents file descriptor leakage

See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041

Descriptors except 0, 1, 2 are closed in childprocesses

Prevents file descriptor leakage

See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041

Example: shared pipe to communicate data between two processes

Descriptors except 0, 1, 2 are closed in childprocesses

Prevents file descriptor leakage

See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041

Example: shared pipe to communicate data between two processes

Use IO#close_on_exec = false

reader, writer = IO.pipe writerfd = writer.fileno my_env["WRITER_FD"] = writerfd.to_s++ writer.close_on_exec = false+ pid = Kernel.spawn(my_env, "ruby #{SERVER_PATH}VixDiskLibServer.rb", [:out, :err] => [LOG_FILE, "a"], :unsetenv_others => true,

String#lines returns an array

Also String#chars, #bytes and #codepoints

Previously, returned enumerators

Above are deprecated for StringIO, IO and friends

String#lines returns an array

Also String#chars, #bytes and #codepoints

Previously, returned enumerators

Above are deprecated for StringIO, IO and friends

StringIO#lines deprecated, so use #each_line instead:

@log_stream.rewind- lines = @log_stream.lines.to_a+ lines = @log_stream.each_line.to_a lines.length.should == 1 line = lines.first.chomp

String#lines returns an array

Also String#chars, #bytes and #codepoints

Previously, returned enumerators

Above are deprecated for StringIO, IO and friends

StringIO#lines deprecated, so use #each_line instead:

@log_stream.rewind- lines = @log_stream.lines.to_a+ lines = @log_stream.each_line.to_a lines.length.should == 1 line = lines.first.chomp

See Pull #714

Ok, great, but are we there yet???

"Fall cleanup" of old code

"...Now I've only been an OpenBSD developer for 11 years, one year less thanthis header has existed, but in that brief time, I've learned a thing or twoabout deleting obsolete code. It doesn't delete itself. And worse, peoplewill continue using it until you force them onto a better path."

http://freshbsd.org/commit/openbsd/68dc781944a2c5b90f8b6e1069a4201750c67f94

"Fall cleanup" of old code

"...Now I've only been an OpenBSD developer for 11 years, one year less thanthis header has existed, but in that brief time, I've learned a thing or twoabout deleting obsolete code. It doesn't delete itself. And worse, peoplewill continue using it until you force them onto a better path."

http://freshbsd.org/commit/openbsd/68dc781944a2c5b90f8b6e1069a4201750c67f94

Git is our friend if we really want it back

"Fall cleanup" of old code

Path to ruby 2.0...

"Fall cleanup" of old code

Path to ruby 2.0...

83 commits

565 lines added

5,553 lines deleted

"Fall cleanup" of old code

Path to ruby 2.0...

83 commits

565 lines added

5,553 lines deleted

A good start?

"Fall cleanup" of old code

More "opportunities"

host directory

soap4r/actionwebservice (fork)

handsoap (fork) - note, useful but still forked :-(

ruport (fork)

ziya (fork) - patches rails!

prototype

old rails plugins

old gems

old monkey patches

Slow tests

Tests take 30+ minutes on CI servers

Separate tests

Minimizing setup (database inserts)

Remove invalid/not useful/duplicate tests

Allocation tracing with ruby 2.1

Cut support for 1.9.3 when 2.0 is stable

Building 2.0 appliances

Verified Ruby 2.0 on CentOS appliance:

Appliance startup

SmartState Analysis "fleecing" using vddk

vCenter inventory

Basic reporting

Building 2.0 appliances

Goal: automate building ruby 2.0 appliances

We currently use ruby 1.9.3 through SCL rpms

Not multi-platform

Restricts updating of some gems

Ruby 2.1 is not yet packaged as SCL rpms

Building 2.0 appliances

Solution:

rpms for base CentOS OS

rpms needed to build ruby and compiled gems

libxml2-devel, libxslt-devel, etc.

Building 2.0 appliances

Solution:

rpms for base CentOS OS

rpms needed to build ruby and compiled gems

libxml2-devel, libxslt-devel, etc.

Use ruby-install or ruby-build for building ruby

https://github.com/postmodern/ruby-install

https://github.com/sstephenson/ruby-build

Building 2.0 appliances

Solution:

rpms for base CentOS OS

rpms needed to build ruby and compiled gems

libxml2-devel, libxslt-devel, etc.

Use ruby-install or ruby-build for building ruby

https://github.com/postmodern/ruby-install

https://github.com/sstephenson/ruby-build

Let bundler handle what it does well...

Developer setup

2.0.0 is not much different from 1.9.3:

Install using rvm, ruby-install, or ruby-build

Manage with rvm, rbenv, or chruby

Need guinea pigs to try it and document any issues

Others tools, such as rubymine, may require some configuration

Links

2.0 open issues: https://github.com/ManageIQ/manageiq/labels/ruby%202

2.0 closed issues: https://github.com/ManageIQ/manageiq/issues?q=label%3A%22ruby+2%22+is%3Aclosed

2.0 in depth: http://globaldev.co.uk/2013/03/ruby-2-0-0-in-detail/

2.1 in depth: http://globaldev.co.uk/2014/05/ruby-2-1-in-detail/

Slides available here: https://github.com/jrafanie/manageiq_summit_ruby20

Slides written in markdown using remarkjs: http://remarkjs.com/#1

uestions?