How does MATLAB benchmark work?

by Forrest Sheng Bao http://fsbao.net

I got a new iMac, the latest 24-inch one with 4G DDR3 memory, 2.66 G Intel Core 2 Duo CPU and NVIDIA GeForce 9400M graphic card.

So, I plan to compare the performance of the same box on Mac OS 10.5 and Ubuntu Linux 9.04 Beta (2.6.28-11 kernel) respectively. The first stuff I tried is MATLAB R2009a.

But I got quite confused regarding how MATLAB benchmark works. I typed bench on MATLAB shell 3 times sequentially, and got following results:
0.1471    0.3886    0.1981    0.3178    0.1815    0.0053
0.0918 0.1435 0.1597 0.2413 0.1659 0.0026
0.0805 0.1395 0.1601 0.2381 0.1633 0.0027
I noticed the huge different among them. So I did a bench(10), which will run the benchmark 10 times. I got this:
0.0939    0.1413    0.1603    0.2455    0.1436    0.0044
0.1106 0.1409 0.1594 0.2379 0.1715 0.0030
0.1145 0.1559 0.1601 0.3205 0.1415 0.0026
0.0584 0.1426 0.1601 0.2354 0.1456 0.0044
0.0799 0.1446 0.1607 0.2784 0.1354 0.0044
0.0900 0.1385 0.1597 0.2375 0.1468 0.0044
0.0983 0.1597 0.1606 0.2851 0.1533 0.0027
0.0770 0.1445 0.1619 0.2373 0.1629 0.0044
0.0810 0.1466 0.1651 0.2493 0.1217 0.0027
0.0892 0.1402 0.1605 0.2367 0.1697 0.0028
This drove me crazy. Let MATLAB do me a favor.
>> min(result(:,1))

ans =

0.0584

>> max(result(:,1))

ans =

0.1145
Still not clear?
>> (max(result(:,1))-min(result(:,1)))/max(result(:,1))

ans =

0.4902
Or, you want a more professional answer?
>> std(result(:,1))/mean(result(:,1))

ans =

0.1852
See the problem? The result equals to this story: Sometimes I need 1 day to build Rome while sometimes I need 2 days to build Rome.

The same machine, the same Linux kernel, the same version of MATLAB composed different stories for us.

The six columns of benchmarking result are of these six tasks:
    LU       LAPACK.                  Floating point, regular memory access.
FFT Fast Fourier Transform. Floating point, irregular memory access.
ODE Ordinary diff. eqn. Data structures and M-files.
Sparse Solve sparse system. Mixed integer and floating point.
2-D plot(fft(eye)). 2-D line drawing graphics.
3-D MathWorks logo. 3-D animated OpenGL graphics.
Please ignore the last two columns of my result since I enabled -nojvm option when starting MATLAB. Thus, I didn't actually plot.

No comments: