Benchmarks: Ruby 1.8.6 and Ruby 1.9.2

using:

Old
Ruby 1.8.6 / Patchlevel 388 (2009-09-09) [i686-darwin9.8.0]
New
Ruby 1.9.2dev / Revision 25094 (2009-09-26) [i386-darwin9.8.0]
Compiler
GCC 4.3.4 -O3 -march=core2 --enable-pthread
System
Mac OS X 10.5.8 on MacBook (2.4GHz Intel Core 2 Duo, 4GB 667 MHz DDR2 SDRAM)

Please note that these are mostly synthetic benchmarks!

Benchmark old new (in seconds)
app_answer 0.500.11
4.5x (-78 %)
app_erb 1.750.99
1.8x (-43 %)
app_factorial[Ruby 1.8.6 broken: stack level too deep]
app_fib 6.241.51
4.1x (-76 %)
app_mandelbrot 1.970.48
4.1x (-76 %)
app_pentomino 84.3445.47
1.9x (-46 %)
app_raise 1.901.86
1.0x (-2 %)
app_strconcat 1.230.69
1.8x (-44 %)
app_tak 8.102.29
3.5x (-72 %)
app_tarai 6.261.88
3.3x (-70 %)
app_uri 4.972.06
2.4x (-59 %)
io_file_create 1.321.78
0.7x (+34 %)
io_file_read 1.020.94
1.1x (-8 %)
io_file_write 0.600.39
1.5x (-35 %)
loop_for 2.663.62
0.7x (+36 %)
loop_generator 18.501.28
14.4x (-93 %)
loop_times 3.593.14
1.1x (-13 %)
loop_whileloop 9.161.58
5.8x (-83 %)
loop_whileloop2 1.840.50
3.7x (-73 %)
so_ackermann[Ruby 1.8.6 broken: stack level too deep]
so_array 5.454.01
1.4x (-26 %)
so_binary_trees 2.560.76
3.4x (-70 %)
so_concatenate 1.520.91
1.7x (-40 %)
so_count_words 0.630.56
1.1x (-12 %)
so_exception 3.182.86
1.1x (-10 %)
so_fannkuch 59.5940.02
1.5x (-33 %)
so_fasta 8.625.08
1.7x (-41 %)
so_k_nucleotide 4.493.20
1.4x (-29 %)
so_lists 0.980.89
1.1x (-9 %)
so_mandelbrot 24.9211.26
2.2x (-55 %)
so_matrix 1.680.99
1.7x (-41 %)
so_meteor_contest 25.8211.50
2.2x (-55 %)
so_nbody 18.128.05
2.3x (-56 %)
so_nested_loop 4.632.87
1.6x (-38 %)
so_nsieve 22.116.48
3.4x (-71 %)
so_nsieve_bits 39.297.07
5.6x (-82 %)
so_object 4.841.99
2.4x (-59 %)
so_partial_sums 19.6110.17
1.9x (-48 %)
so_pidigits 3.802.66
1.4x (-30 %)
so_random 1.760.61
2.9x (-65 %)
so_reverse_complement 2.963.03
1.0x (+2 %)
so_sieve 0.650.33
2.0x (-49 %)
so_spectralnorm 23.407.64
3.1x (-67 %)
vm1_block* 10.374.25
2.4x (-59 %)
vm1_const* 5.681.51
3.8x (-73 %)
vm1_ensure* 6.410.23
28.5x (-96 %)
vm1_ivar* 5.111.61
3.2x (-69 %)
vm1_ivar_set* 5.771.61
3.6x (-72 %)
vm1_length* 8.482.29
3.7x (-73 %)
vm1_neq* 5.931.92
3.1x (-68 %)
vm1_not* 1.920.98
1.9x (-49 %)
vm1_rescue* 2.450.20
12.0x (-92 %)
vm1_simplereturn* 10.252.84
3.6x (-72 %)
vm1_swap* 10.631.23
8.6x (-88 %)
vm2_array* 2.921.73
1.7x (-41 %)
vm2_case* 2.170.06
34.0x (-97 %)
vm2_eval* 10.2840.07
0.3x (+290 %)
vm2_method* 9.603.98
2.4x (-59 %)
vm2_mutex* 2.273.05
0.7x (+35 %)
vm2_poly_method* 13.015.25
2.5x (-60 %)
vm2_poly_method_ov* 2.010.21
9.4x (-89 %)
vm2_proc* 3.571.34
2.7x (-62 %)
vm2_regexp* 3.402.53
1.3x (-26 %)
vm2_send* 1.980.49
4.0x (-75 %)
vm2_super* 2.540.86
2.9x (-66 %)
vm2_unif1* 1.560.40
3.9x (-74 %)
vm2_zsuper* 2.970.96
3.1x (-68 %)
vm3_gc 1.311.63
0.8x (+25 %)
vm3_thread_create_join 0.343.82
0.1x (+1017 %)
vm3_thread_mutex 65.4175.50
0.9x (+15 %)
epic_battle 44.4427.77
1.6x (-38 %)
logbench 2.451.32
1.9x (-46 %)
zahlen 2.971.67
1.8x (-44 %)
test_rails_app 8.325.06
1.6x (-39 %)
fukubukuro 28.7513.10
2.2x (-54 %)
simple_active_record 10.7810.34
1.0x (-4 %)
rake_startup 0.740.45
1.6x (-39 %)
lib_coderay 3.093.08
1.0x (-0 %)
lib_coderay_test 33.6632.50
1.0x (-3 %)
lib_rspec 2.742.14
1.3x (-22 %)
interpreter_start 0.010.02
0.5x (+100 %)
interpreter_require 0.040.07
0.6x (+68 %)

* means that results of loop_whileloop/loop_whileloop2 are subtracted.

group # speedup
Core (vm)27 5.4x
Loops (loop)5 5.2x
Mini apps (app)10 2.8x
Shootout (so)23 2.1x
Other (other)7 1.7x
I/O (io)3 1.1x
Libraries (lib)3 1.1x
Interpreter (interpreter)2 0.5x

Sourcecode

require 'cgi'

puts <<-HEAD
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" version="-//W3C//DTD XHTML 1.1//EN" xml:lang="de">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
  <title>Ruby 1.8 vs. 1.9 Benchmarks</title>
  <style type="text/css" media="screen">
    html { font-family: 'Trebuchet MS', Verdana, Arial, sans-serif; }
    td {
      vertical-align: middle;
      white-space: nowrap;
      padding: 0.1em 0.3em;
      text-align: left;
      border: 1px solid black; border-width: 1px 0;
    }
    table {
      border-collapse: collapse;
      margin: 0;
    }
    table h6 {
      margin: 0.1em; padding: 0.1em 0.3em;
      white-space: nowrap;
      font-size: 0.7em;
    }
    thead td { border-top-width: 0; }
    .faster { background-color: #2f2; }
    .slower { background-color: #f22; }
    .broken { background-color: silver; }
    .time { font-size: 0.9em; }
    .right { text-align: right; }
    .note { color: red; font-weight: bold; }
    dd { font-style: italic; }
  </style>
</head>
<body>

<h1>Benchmarks: Ruby 1.8.6 and Ruby 1.9.2</h1>

<p>using:</p>

<dl>
  <dt>Old</dt>
  <dd>Ruby 1.8.6 / Patchlevel 388 (2009-09-09) [i686-darwin9.8.0]</dd>
  <dt>New</dt>
  <dd>Ruby 1.9.2dev / Revision 25094 (2009-09-26) [i386-darwin9.8.0]</dd>
  <dt>Compiler</dt>
  <dd>GCC 4.3.4 -O3 -march=core2 --enable-pthread</dd>
  <dt>System</dt>
  <dd>Mac OS X 10.5.8 on MacBook (2.4GHz Intel Core 2 Duo, 4GB 667 MHz DDR2 SDRAM)</dd>
</dl>

<p class="note">
  Please note that these are mostly synthetic benchmarks!
</p>

<table>
  <thead>
    <tr>
      <td class="benchmark">Benchmark</td>
      <td class="time right">old</td>
      <td class="time right">new</td>
      <td class="time">(in seconds)</td>
    </tr>
  </thead>
  <tbody>
HEAD

WIDTH_FACTOR = 500
MAX_WIDTH = WIDTH_FACTOR
EXPLAIN = {
  'app' => 'Mini apps',
  'so' => 'Shootout',
  'lib' => 'Libraries',
  'vm' => 'Core',
  'loop' => 'Loops',
  'io' => 'I/O',
  'other' => 'Other',
  'interpreter' => 'Interpreter',
}

speedups_for = Hash.new { |h, k| h[k] = [] }
looptime = Hash.new([0, 0])


DATA.read.scan(/^ *(\w+\*?)\s+(\S+)\s+(\S+) *(.+)?/) do |name, old, new, broken|
  if broken
    puts "<tr class=\"broken\">"
    puts "<td class=\"benchmark\">%s</td>" % [name] + "<td></td>" * 2 + "<td>[#{$4}]</td>"
    puts "</tr>"
    next
  end
    
  old = old.to_f
  new = new.to_f
  
  case name
  when /^loop_whileloop(2)?$/
    looptime[$1 || '1'] = [old, new]
  when false #/^vm([12])/
    offset_old, offset_new = looptime[$1]
    new -= offset_new
    old -= offset_old
    name << '*'
  end
  
  # build HTML table
  puts "<tr>"
  puts '<td class="benchmark">%s</td>' % name
  puts '<td class="time right">%0.2f</td><td class="time right">%0.2f</td>' %
    [old, new]
  width = [WIDTH_FACTOR * ((new / old) - 1).abs, MAX_WIDTH].min
  puts '<td><h6 style="width: %dpx" class="%s">%0.1fx (%+0.0f %%)</h6></td>' % [
    width,
    (new <= old ? 'faster' : 'slower'),
    (old / new),
    100 * ((new / old) - 1)
  ]
  puts "</tr>"
  
  speedup = (old / new)
  if name[/^(app|loop|so|vm|lib|io|interpreter)/]
    speedups_for[$1] << speedup
  else
    speedups_for['other'] << speedup
  end
end
puts "</tbody></table>"

puts "<p><small>* means that results of loop_whileloop/loop_whileloop2 are subtracted.</small></p>"

puts <<-HTML
<table>
  <thead>
    <tr>
      <td class="benchmark">group</td>
      <td class="right">#</td>
      <td class="right">speedup</td>
    </tr>
  </thead>
  <tbody>
HTML

speedups_for.map do |tag, speedups|
  average_speedup = speedups.inject { |sum,x| sum + x } / speedups.size
  [average_speedup, tag, speedups]
end.sort.reverse.each do |average_speedup, tag, speedups|
  puts "<tr><td>#{EXPLAIN[tag]} (#{tag})</td><td class=\"right\">%d</td>" % [speedups.size]
  puts "<td class=\"right\">%0.1fx</td></tr>" % [average_speedup]
end
puts "</tbody></table>"

puts <<-BODY
<h3>Sourcecode</h3>
<pre>#{DATA.rewind; CGI.escapeHTML(DATA.read)}</pre>
</body>
</html>
BODY
__END__
            app_answer            0.503           0.112
               app_erb            1.754           0.994
         app_factorial            0.578           0.315  Ruby 1.8.6 broken: stack level too deep
               app_fib            6.242           1.514
        app_mandelbrot            1.966           0.478
         app_pentomino           84.343          45.467
             app_raise            1.899           1.856
         app_strconcat            1.234           0.695
               app_tak            8.103           2.285
             app_tarai            6.257           1.884
               app_uri            4.970           2.062
        io_file_create            1.324           1.779
          io_file_read            1.019           0.935
         io_file_write            0.597           0.386
              loop_for            2.662           3.625
        loop_generator           18.498           1.284
            loop_times            3.591           3.141
        loop_whileloop            9.156           1.581
       loop_whileloop2            1.837           0.499
          so_ackermann           10.594           1.812  Ruby 1.8.6 broken: stack level too deep
              so_array            5.447           4.008
       so_binary_trees            2.562           0.763
        so_concatenate            1.519           0.906
        so_count_words            0.634           0.558
          so_exception            3.182           2.858
           so_fannkuch           59.590          40.023
              so_fasta            8.620           5.077
       so_k_nucleotide            4.494           3.203
              so_lists            0.976           0.888
         so_mandelbrot           24.922          11.264
             so_matrix            1.676           0.993
     so_meteor_contest           25.815          11.501
              so_nbody           18.119           8.049
        so_nested_loop            4.632           2.866
             so_nsieve           22.108           6.477
        so_nsieve_bits           39.291           7.065
             so_object            4.838           1.986
       so_partial_sums           19.607          10.173
           so_pidigits            3.797           2.661
             so_random            1.765           0.613
 so_reverse_complement            2.962           3.030
              so_sieve            0.652           0.330
       so_spectralnorm           23.402           7.643
             vm1_block*          10.374           4.246
             vm1_const*           5.678           1.506
            vm1_ensure*           6.406           0.225
              vm1_ivar*           5.110           1.606
          vm1_ivar_set*           5.766           1.607
            vm1_length*           8.481           2.292
               vm1_neq*           5.932           1.923
               vm1_not*           1.919           0.985
            vm1_rescue*           2.449           0.204
      vm1_simplereturn*          10.247           2.838
              vm1_swap*          10.634           1.230
             vm2_array*           2.918           1.729
              vm2_case*           2.175           0.064
              vm2_eval*          10.275          40.066
            vm2_method*           9.599           3.976
             vm2_mutex*           2.265           3.047
       vm2_poly_method*          13.013           5.249
    vm2_poly_method_ov*           2.014           0.215
              vm2_proc*           3.569           1.344
            vm2_regexp*           3.402           2.531
              vm2_send*           1.978           0.489
             vm2_super*           2.537           0.864
             vm2_unif1*           1.558           0.398
            vm2_zsuper*           2.972           0.962
                vm3_gc            1.308           1.630
vm3_thread_create_join            0.342           3.821
      vm3_thread_mutex           65.408          75.505
      
           epic_battle           44.443          27.765
              logbench            2.448           1.323
                zahlen            2.970           1.670
        test_rails_app            8.322           5.061
            fukubukuro           28.747          13.104
  simple_active_record           10.783          10.341
          rake_startup            0.738           0.451
           lib_coderay            3.094           3.083
      lib_coderay_test           33.659          32.496
             lib_rspec            2.742           2.137
     interpreter_start            0.009           0.018
   interpreter_require            0.044           0.074