How does MATLAB benchmark work?

by Forrest Sheng Bao http://fsbao.net

I got a new iMac, the latest 24-inch one with 4G DDR3 memory, 2.66 G Intel Core 2 Duo CPU and NVIDIA GeForce 9400M graphic card.

So, I plan to compare the performance of the same box on Mac OS 10.5 and Ubuntu Linux 9.04 Beta (2.6.28-11 kernel) respectively. The first stuff I tried is MATLAB R2009a.

But I got quite confused regarding how MATLAB benchmark works. I typed bench on MATLAB shell 3 times sequentially, and got following results:

0.1471    0.3886    0.1981    0.3178    0.1815    0.0053
0.0918    0.1435    0.1597    0.2413    0.1659    0.0026
0.0805    0.1395    0.1601    0.2381    0.1633    0.0027

I noticed the huge different among them. So I did a bench(10), which will run the benchmark 10 times. I got this:

0.0939    0.1413    0.1603    0.2455    0.1436    0.0044
0.1106    0.1409    0.1594    0.2379    0.1715    0.0030
0.1145    0.1559    0.1601    0.3205    0.1415    0.0026
0.0584    0.1426    0.1601    0.2354    0.1456    0.0044
0.0799    0.1446    0.1607    0.2784    0.1354    0.0044
0.0900    0.1385    0.1597    0.2375    0.1468    0.0044
0.0983    0.1597    0.1606    0.2851    0.1533    0.0027
0.0770    0.1445    0.1619    0.2373    0.1629    0.0044
0.0810    0.1466    0.1651    0.2493    0.1217    0.0027
0.0892    0.1402    0.1605    0.2367    0.1697    0.0028

This drove me crazy. Let MATLAB do me a favor.

>> min(result(:,1))

ans =

0.0584

>> max(result(:,1))

ans =

0.1145

Still not clear?

>> (max(result(:,1))-min(result(:,1)))/max(result(:,1))

ans =

0.4902

Or, you want a more professional answer?

>> std(result(:,1))/mean(result(:,1))

ans =

0.1852

See the problem? The result equals to this story: Sometimes I need 1 day to build Rome while sometimes I need 2 days to build Rome.

The same machine, the same Linux kernel, the same version of MATLAB composed different stories for us.

The six columns of benchmarking result are of these six tasks:

    LU       LAPACK.                  Floating point, regular memory access.
 FFT      Fast Fourier Transform.  Floating point, irregular memory access.
 ODE      Ordinary diff. eqn.      Data structures and M-files.
 Sparse   Solve sparse system.     Mixed integer and floating point.
 2-D      plot(fft(eye)).          2-D line drawing graphics.
 3-D      MathWorks logo.          3-D animated OpenGL graphics.

Please ignore the last two columns of my result since I enabled -nojvm option when starting MATLAB. Thus, I didn't actually plot.