Notes on testing

A couple of notes on the performance tests I've been doing.


First in all of them I have been doing sums groups of four, accessing members of the class eight times. For example



for (iTrial = 0; iTrial < kNumTrials; iTrial++) {
    v = aData[iTrial];
    iSum += v.iX + v.iY;
    iSum += v.iX + v.iY;
    iSum += v.iX + v.iY;
    iSum += v.iX + v.iY;
}


instead of



for (iTrial = 0; iTrial < kNumTrials; iTrial++) {
    v = aData[iTrial];
    iSum += v.iX;
}

This is as the latter loop consists mostly of loop commands, not the code that's being tested (the last line). Replacing that with four instances of iSum += v.iX + v.iY means it spends proportionately more time doing what the performance tests are trying to test, so the test results are more useful. It's also more like the code where such optimisations become important: code with many accesses of members from the same object, in a loop or a frequently called function.


Second I've been reporting the relative speedups but not the actual results of tests. This is as the absolute times are not that important, and anyway are not fixed: they will vary significantly between different users. They are also pretty arbitrary as when testing I set the number of trials to give a long enough time that Flash's millisecond timer can measure it, as well as rewriting the code to show exactly what I want.