Code benchmark on Windows

[Biterider]

Hi
I spent some time adapting the code from this thread http://masm32.com/board/index.php?topic=10099.msg110903#msg110903 to run on Windows.
The code was originally developed for Linux and adapted for UEFI and now further developed for Windows.

The intent was to see if such a benchmark was feasible and gave reproducible results.
Looking at the numbers and comparing them to the UEFI version, the code execution time looks very similar and the coefficient of determination R2 looks remarkably good. 

If you want to test it, the code is on Github and the binary in the attachment. It requires DebugCenter to get the results.

Biterider

[HSE]

Hi Biterider!!

Fantastic! Very simple to test little algorithms.

Code Select Expand

Spurious =    16          [-]         Number of Spurious min. Values
MaxMaxDiff =  84868       [Tick]      Max. Spread of the Values
MeanVar =     4.166E+0005 [Tick^2]    Mean Variance
VarOfVars =   1.214E+0012 [Tick^4]    Variance of Variances
MinsMSE =     5.546E+0002 [Tick^2]    Linear Regression MSE
R2 =          9.973E-0001 [-]         Determination Coeficient
Overhead =    78.4        [Tick]      Measurement Overhead
Measure =     15.5        [Tick]      Code Execution Time

the coefficient of determination R2 looks remarkably good. 

You can't have a bad R², forget that  . You have to see variance, and Variance of Variances is at list 10⁶ times bigger than microLinux or UEFI.


Don't look very good at all:

Code (Whitout loop) Select Expand

Spurious =   9           [-]           Number of Spurious min. Values
MaxMaxDiff =  89794        [Tick]     Max. Spread of the Values
MeanVar =     1.005E+0005 [Tick^2]    Mean Variance
VarOfVars =    4.369E+0010 [Tick^4]  Variance of Variances
MinsMSE =     2.145E+0000 [Tick^2]    Linear Regression MSE
R2 =          8.739E-0003 [-]         Determination Coeficient
Overhead =   101.4       [Tick]      Measurement Overhead
Measure =     0.0         [Tick]       Code Execution Time


Code (With loop but without target) Select Expand

Spurious =   1           [-]           Number of Spurious min. Values
MaxMaxDiff =  88474        [Tick]     Max. Spread of the Values
MeanVar =     1.095E+0005 [Tick^2]    Mean Variance
VarOfVars =    6.304E+0010 [Tick^4]  Variance of Variances
MinsMSE =     1.918E+0001 [Tick^2]    Linear Regression MSE
R2 =          9.694E-0001 [-]         Determination Coeficient
Overhead =   100.6       [Tick]      Measurement Overhead
Measure =     0.9         [Tick]       Code Execution Time


Code (With loop and target) Select Expand

Spurious =   2           [-]           Number of Spurious min. Values
MaxMaxDiff =  95388        [Tick]     Max. Spread of the Values
MeanVar =     3.608E+0005 [Tick^2]    Mean Variance
VarOfVars =    2.637E+0011 [Tick^4]  Variance of Variances
MinsMSE =     1.112E+0002 [Tick^2]    Linear Regression MSE
R2 =          9.994E-0001 [-]         Determination Coeficient
Overhead =   82.7         [Tick]       Measurement Overhead
Measure =     15.4       [Tick]      Code Execution Time


HSE

[Biterider]

Hi HSE
I agree with you that the "quality" of the measurement in Windows is not as good as in the other environments mentioned.
The question we need to answer is whether it is good enough for code timing.

The criteria presented by Paolini and the method he showed seem to me to be much better than the simple and fallacious start-stop measurement method.
Repeatability and linearity are the criteria for me to validate the method. Variance, variance of variances and so on are parameters that define accuracy and dispersion.
The last parameter is much worse under Windows, as expected, but still the approach is reasonable and well applicable.

Just my 2 cents 

Biterider

[HSE]

Hi Biterider!

The criteria presented by Paoloni and the method he showed seem to me to be much better than the simple and fallacious start-stop measurement method.

No doubt.

Repeatability and linearity are the criteria for me to validate the method.

Not for Paoloni. Repeatability is just an expression of measurement variance, and that is the thing. But results are very stable because are processed sample groups, not individual samples 

Variance, variance of variances and so on are parameters that define accuracy and dispersion.

Exatly this is what make a method reliable or not for Paoloni.

Just my 2 cents 

A complete piece.

I think a more complex design is necesary because I see little strange things, also in UEFI, perhaps related to core speed variations. Probably that is a processor response to load or temperature. But that is a complete speculation 

[Biterider]

Hi
I took a closer look at the data from the benchmark application and found an issue that also occurs in other related projects.

The problem lies in the linear regression, which assumes that the first data is at X=0, but in reality it is at X=1.
For this reason, the offset, called overhead in this algorithm, needs to be corrected by one slope unit.

I have made the corrections in the Github projects, but those who use the algorithm must also fix their code.

Biterider

[HSE]

Hi Biterider!

The problem lies in the linear regression, which assumes that the first data is at X=0, but in reality it is at X=1.

No. You begin with x=0:

Code Select Expand

    xor rsi, rsi
         ···
    xor edi, edi
        ···
    mov ebx, esi


Just in the slope there is an extra because:

Code Select Expand

    dec ebx

HSE

[Biterider]

Hi HSE
No, it is like I said.
The index starts with zero but for the following calculations you have to think of it as X=1.
That is why you have to correct the offset/overhead value, otherwise you will get the offset at X=1 and that is not what you want.
The axis intersection is at X=0.


Biterider

[HSE]

Hi

No..

No...

What I can say? With same code, here begin with x=0  :

Code Select Expand

SIZE_OF_STAT  equ 2
BOUND_OF_LOOP equ 3


Code Select Expand

ebx = 0 [BenchmarkWin.asm, 127]
ebx = 0 [BenchmarkWin.asm, 127]
ebx = 1 [BenchmarkWin.asm, 127]
    testing opcodes [BenchmarkWin.asm, 130]
ebx = 1 [BenchmarkWin.asm, 127]
    testing opcodes [BenchmarkWin.asm, 130]
ebx = 2 [BenchmarkWin.asm, 127]
    testing opcodes [BenchmarkWin.asm, 130]
    testing opcodes [BenchmarkWin.asm, 130]
ebx = 2 [BenchmarkWin.asm, 127]
    testing opcodes [BenchmarkWin.asm, 130]
    testing opcodes [BenchmarkWin.asm, 130]


[Biterider]

Hi HSE
Sorry, I think I was wrong. 
The first iteration does not execute the test code and as such, it is evaluated at X=0.
I'll have to roll back the code.


Biterider

[HSE]