3.2GHz and 3.6GHz, no big difference?

I had an interesting discovery about my AMD Phenom II X6 1090T CPU. I threw a very CPU intensive job to it. But the time (measured in Linux command ``time'') spent to finish the job was the pretty much the same when the CPU works at 3.6 GHz and 3.2 GHz (I played a trick in AMD Cool'n'Quite setting in my BIOS and verified the frequency for the core running my job by cat /proc/cpuinfo | grep -i "cpu Mhz").

So, I am wondering why. I do not think it is because of the bottleneck at the memory. My program only uses 3% of the memory which in total is 8 GB.