Benchmarks: Ruby 1.8 and Ruby 1.9

using:

Old
Ruby 1.8.6 / Patchlevel 113 (2007-12-03)
New
Ruby 1.9.0 / Revision 15006 (2008-01-12)
System
Mac OS X 10.5.1 on First Generation MacBook (1.83GHz Intel Core Duo, 2GB 667 MHz DDR2 SDRAM)

Please note that these are mostly synthetic benchmarks!

Benchmark old new (in seconds)
app_answer 0.580.15
4.0x (-75 %)
app_erb 2.192.05
1.1x (-6 %)
app_factorial[Ruby 1.8.6 broken (stack level too deep)]
app_fib 7.511.98
3.8x (-74 %)
app_mandelbrot 2.441.10
2.2x (-55 %)
app_pentomino 107.9459.81
1.8x (-45 %)
app_raise 2.492.84
0.9x (+14 %)
app_strconcat 1.801.18
1.5x (-35 %)
app_tak 9.662.72
3.6x (-72 %)
app_tarai 7.742.14
3.6x (-72 %)
app_uri 4.672.60
1.8x (-44 %)
io_file_create 1.852.42
0.8x (+30 %)
io_file_read 1.161.93
0.6x (+67 %)
io_file_write 0.800.68
1.2x (-15 %)
interpreter_launch 0.010.04
0.3x (+207 %)
loop_generator 24.571.78
13.8x (-93 %)
loop_times 5.904.46
1.3x (-24 %)
loop_whileloop 10.411.91
5.5x (-82 %)
loop_whileloop2 2.080.38
5.5x (-82 %)
so_ackermann[Ruby 1.8.6 broken (stack level too deep)]
so_array 7.754.51
1.7x (-42 %)
so_binary_trees 3.341.56
2.1x (-53 %)
so_concatenate 2.191.33
1.7x (-39 %)
so_count_words 1.558.42
0.2x (+442 %)
so_exception 4.244.02
1.1x (-5 %)
so_fannkuch 78.6960.23
1.3x (-23 %)
so_fasta 10.767.41
1.5x (-31 %)
so_k_nucleotide 6.205.74
1.1x (-8 %)
so_lists 1.351.10
1.2x (-18 %)
so_mandelbrot 33.0119.18
1.7x (-42 %)
so_matrix 2.301.23
1.9x (-46 %)
so_meteor_contest 37.3423.41
1.6x (-37 %)
so_nbody 23.9812.72
1.9x (-47 %)
so_nested_loop 6.053.96
1.5x (-35 %)
so_nsieve 23.458.46
2.8x (-64 %)
so_nsieve_bits 52.2510.41
5.0x (-80 %)
so_object 7.022.72
2.6x (-61 %)
so_partial_sums 27.2913.36
2.0x (-51 %)
so_pidigits 5.967.02
0.8x (+18 %)
so_random 2.220.79
2.8x (-64 %)
so_reverse_complement 3.894.52
0.9x (+16 %)
so_sieve 0.690.23
3.1x (-67 %)
so_spectralnorm 28.1814.62
1.9x (-48 %)
vm1_block 16.045.50
2.9x (-66 %)
vm1_const 6.750.90
7.5x (-87 %)
vm1_ensure 7.160.04
183.6x (-99 %)
vm1_ivar 6.564.13
1.6x (-37 %)
vm1_ivar_set 6.834.63
1.5x (-32 %)
vm1_length 9.992.97
3.4x (-70 %)
vm1_neq 8.231.57
5.2x (-81 %)
vm1_not 2.480.72
3.4x (-71 %)
vm1_rescue 3.490.23
14.9x (-93 %)
vm1_simplereturn 11.953.23
3.7x (-73 %)
vm1_swap 15.260.74
20.7x (-95 %)
vm2_array 3.882.88
1.3x (-26 %)
vm2_case 2.860.42
6.8x (-85 %)
vm2_eval 13.5958.71
0.2x (+332 %)
vm2_method 11.164.98
2.2x (-55 %)
vm2_mutex 3.343.83
0.9x (+15 %)
vm2_poly_method 15.186.59
2.3x (-57 %)
vm2_poly_method_ov 2.630.65
4.1x (-75 %)
vm2_proc 5.471.99
2.7x (-64 %)
vm2_regexp 3.003.79
0.8x (+26 %)
vm2_send 2.420.73
3.3x (-70 %)
vm2_super 3.111.29
2.4x (-59 %)
vm2_unif1 2.060.68
3.0x (-67 %)
vm2_zsuper 3.671.34
2.7x (-63 %)
vm3_thread_create_join 0.535.13
0.1x (+864 %)
vm3_thread_mutex 78.3846.37
1.7x (-41 %)
lib_coderay 3.123.13
1.0x (+1 %)
lib_json_pure 0.370.28
1.3x (-24 %)
lib_rdoc_coderay 4.296.02
0.7x (+40 %)
interpreter_start 0.010.02
0.3x (+186 %)
interpreter_require 0.050.11
0.5x (+107 %)

* means that results of loop_whileloop/loop_whileloop2 are subtracted.

group # speedup
Core (vm)26 10.9x
Loops (loop)4 6.5x
Mini apps (app)10 2.4x
Shootout (so)23 1.8x
Libraries (lib)3 1.0x
I/O (io)3 0.8x
Interpreter (interpreter)3 0.4x

Sourcecode

require 'cgi'

puts <<-HEAD
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" version="-//W3C//DTD XHTML 1.1//EN" xml:lang="de">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
  <title>Ruby 1.8 vs. 1.9 Benchmarks</title>
  <style type="text/css" media="screen">
    html { font-family: 'Trebuchet MS', Verdana, Arial, sans-serif; }
    td {
      vertical-align: middle;
      white-space: nowrap;
      padding: 0.1em 0.3em;
      text-align: left;
      border: 1px solid black; border-width: 1px 0;
    }
    table {
      border-collapse: collapse;
      margin: 0;
    }
    table h6 {
      margin: 0.1em; padding: 0.1em 0.3em;
      white-space: nowrap;
      font-size: 0.7em;
    }
    thead td { border-top-width: 0; }
    .faster { background-color: #2f2; }
    .slower { background-color: #f22; }
    .broken { background-color: silver; }
    .time { font-size: 0.9em; }
    .right { text-align: right; }
    .note { color: red; font-weight: bold; }
    dd { font-style: italic; }
  </style>
</head>
<body>

<h1>Benchmarks: Ruby 1.8 and Ruby 1.9</h1>

<p>using:</p>

<dl>
  <dt>Old</dt>
  <dd>Ruby 1.8.6 / Patchlevel 113 (2007-12-03)</dd>
  <dt>New</dt>
  <dd>Ruby 1.9.0 / Revision 15006 (2008-01-12)</dd>
  <dt>System</dt>
  <dd>Mac OS X 10.5.1 on First Generation MacBook (1.83GHz Intel Core Duo, 2GB 667 MHz DDR2 SDRAM)</dd>
</dl>

<p class="note">
  Please note that these are mostly synthetic benchmarks!
</p>

<table>
  <thead>
    <tr>
      <td class="benchmark">Benchmark</td>
      <td class="time right">old</td>
      <td class="time right">new</td>
      <td class="time">(in seconds)</td>
    </tr>
  </thead>
  <tbody>
HEAD

WIDTH_FACTOR = 500
MAX_WIDTH = WIDTH_FACTOR
EXPLAIN = {
  'app' => 'Mini apps',
  'so' => 'Shootout',
  'lib' => 'Libraries',
  'vm' => 'Core',
  'loop' => 'Loops',
  'io' => 'I/O',
  'other' => 'Other',
  'interpreter' => 'Interpreter',
}

speedups_for = Hash.new { |h, k| h[k] = [] }
looptime = Hash.new([0, 0])


DATA.read.scan(/^ *(\w+)\*?\s+(\S+)\s+(\S+) *(.+)?/) do |name, old, new, broken|
  if broken
    puts "<tr class=\"broken\">"
    puts "<td class=\"benchmark\">%s</td>" % [name] + "<td></td>" * 2 + "<td>[#{$4}]</td>"
    puts "</tr>"
    next
  end
    
  old = old.to_f
  new = new.to_f
  
  case name
  when /^loop_whileloop(2)?$/
    looptime[$1 || '1'] = [old, new]
  when false #/^vm([12])/
    offset_old, offset_new = looptime[$1]
    new -= offset_new
    old -= offset_old
    name << '*'
  end
  
  # build HTML table
  puts "<tr>"
  puts '<td class="benchmark">%s</td>' % name
  puts '<td class="time right">%0.2f</td><td class="time right">%0.2f</td>' %
    [old, new]
  width = [WIDTH_FACTOR * ((new / old) - 1).abs, MAX_WIDTH].min
  puts '<td><h6 style="width: %dpx" class="%s">%0.1fx (%+0.0f %%)</h6></td>' % [
    width,
    (new <= old ? 'faster' : 'slower'),
    (old / new),
    100 * ((new / old) - 1)
  ]
  puts "</tr>"
  
  speedup = (old / new)
  if name[/^(app|loop|so|vm|lib|io|interpreter)/]
    speedups_for[$1] << speedup
  else
    speedups_for['other'] << speedup
  end
end
puts "</tbody></table>"

puts "<p><small>* means that results of loop_whileloop/loop_whileloop2 are subtracted.</small></p>"

puts <<-HTML
<table>
  <thead>
    <tr>
      <td class="benchmark">group</td>
      <td class="right">#</td>
      <td class="right">speedup</td>
    </tr>
  </thead>
  <tbody>
HTML

speedups_for.map do |tag, speedups|
  average_speedup = speedups.inject { |sum,x| sum + x } / speedups.size
  [average_speedup, tag, speedups]
end.sort.reverse.each do |average_speedup, tag, speedups|
  puts "<tr><td>#{EXPLAIN[tag]} (#{tag})</td><td class=\"right\">%d</td>" % [speedups.size]
  puts "<td class=\"right\">%0.1fx</td></tr>" % [average_speedup]
end
puts "</tbody></table>"

puts <<-BODY
<h3>Sourcecode</h3>
<pre>#{DATA.rewind; CGI.escapeHTML(DATA.read)}</pre>
</body>
</html>
BODY
__END__
            app_answer            0.579           0.146
               app_erb            2.190           2.053
         app_factorial            0.935           1.122  Ruby 1.8.6 broken (stack level too deep)
               app_fib            7.512           1.977
        app_mandelbrot            2.440           1.096
         app_pentomino          107.936          59.806
             app_raise            2.490           2.844
         app_strconcat            1.804           1.181
               app_tak            9.663           2.716
             app_tarai            7.735           2.137
               app_uri            4.668           2.600
        io_file_create            1.853           2.416
          io_file_read            1.155           1.928
         io_file_write            0.800           0.679
    interpreter_launch            0.014           0.043
        loop_generator           24.565           1.783
            loop_times            5.900           4.465
        loop_whileloop           10.408           1.905
       loop_whileloop2            2.081           0.381
          so_ackermann           14.262           2.206  Ruby 1.8.6 broken (stack level too deep)
              so_array            7.754           4.515
       so_binary_trees            3.337           1.561
        so_concatenate            2.189           1.326
        so_count_words            1.555           8.424
          so_exception            4.236           4.018
           so_fannkuch           78.685          60.227
              so_fasta           10.763           7.407
       so_k_nucleotide            6.204           5.736
              so_lists            1.347           1.104
         so_mandelbrot           33.010          19.185
             so_matrix            2.301           1.232
     so_meteor_contest           37.336          23.410
              so_nbody           23.977          12.724
        so_nested_loop            6.051           3.962
             so_nsieve           23.447           8.462
        so_nsieve_bits           52.250          10.413
             so_object            7.018           2.717
       so_partial_sums           27.290          13.360
           so_pidigits            5.956           7.021
             so_random            2.216           0.788
 so_reverse_complement            3.889           4.517
              so_sieve            0.691           0.225
       so_spectralnorm           28.181          14.619
             vm1_block*          16.039           5.496
             vm1_const*           6.748           0.903
            vm1_ensure*           7.159           0.039
              vm1_ivar*           6.559           4.133
          vm1_ivar_set*           6.834           4.635
            vm1_length*           9.990           2.970
               vm1_neq*           8.226           1.571
               vm1_not*           2.479           0.723
            vm1_rescue*           3.486           0.234
      vm1_simplereturn*          11.952           3.225
              vm1_swap*          15.258           0.737
             vm2_array*           3.878           2.879
              vm2_case*           2.863           0.418
              vm2_eval*          13.586          58.712
            vm2_method*          11.159           4.984
             vm2_mutex*           3.337           3.828
       vm2_poly_method*          15.176           6.595
    vm2_poly_method_ov*           2.635           0.646
              vm2_proc*           5.466           1.988
            vm2_regexp*           3.005           3.792
              vm2_send*           2.424           0.734
             vm2_super*           3.108           1.289
             vm2_unif1*           2.055           0.678
            vm2_zsuper*           3.666           1.339
vm3_thread_create_join            0.532           5.130
      vm3_thread_mutex           78.377          46.367
           lib_coderay            3.119           3.135
         lib_json_pure            0.370           0.283
      lib_rdoc_coderay            4.295           6.019
     interpreter_start            0.007           0.020
   interpreter_require            0.054           0.112