News:

Masm32 SDK description, downloads and other helpful links
Message to All Guests

Main Menu

Code benchmark on Windows

Started by Biterider, January 12, 2023, 05:39:00 AM

Previous topic - Next topic

Biterider

Hi
I spent some time adapting the code from this thread http://masm32.com/board/index.php?topic=10099.msg110903#msg110903 to run on Windows.
The code was originally developed for Linux and adapted for UEFI and now further developed for Windows.

The intent was to see if such a benchmark was feasible and gave reproducible results.
Looking at the numbers and comparing them to the UEFI version, the code execution time looks very similar and the coefficient of determination R2 looks remarkably good.  :cool:

If you want to test it, the code is on Github and the binary in the attachment. It requires DebugCenter to get the results.

Biterider

HSE

Hi Biterider!!

:thumbsup: Fantastic! Very simple to test little algorithms.

Spurious =    16          [-]         Number of Spurious min. Values
MaxMaxDiff =  84868       [Tick]      Max. Spread of the Values
MeanVar =     4.166E+0005 [Tick^2]    Mean Variance
VarOfVars =   1.214E+0012 [Tick^4]    Variance of Variances
MinsMSE =     5.546E+0002 [Tick^2]    Linear Regression MSE
R2 =          9.973E-0001 [-]         Determination Coeficient
Overhead =    78.4        [Tick]      Measurement Overhead
Measure =     15.5        [Tick]      Code Execution Time


Quote from: Biterider on January 12, 2023, 05:39:00 AM
the coefficient of determination R2 looks remarkably good.  :cool:

You can't have a bad R2, forget that  :biggrin:. You have to see variance, and Variance of Variances is at list 106 times bigger than microLinux or UEFI.


Don't look very good at all:

Code (Whitout loop) Select
Spurious =   9           [-]           Number of Spurious min. Values
MaxMaxDiff =  89794        [Tick]     Max. Spread of the Values
MeanVar =     1.005E+0005 [Tick^2]    Mean Variance
VarOfVars =    4.369E+0010 [Tick^4]  Variance of Variances
MinsMSE =     2.145E+0000 [Tick^2]    Linear Regression MSE
R2 =          8.739E-0003 [-]         Determination Coeficient
Overhead =   101.4       [Tick]      Measurement Overhead
Measure =     0.0         [Tick]       Code Execution Time


Code (With loop but without target) Select
Spurious =   1           [-]           Number of Spurious min. Values
MaxMaxDiff =  88474        [Tick]     Max. Spread of the Values
MeanVar =     1.095E+0005 [Tick^2]    Mean Variance
VarOfVars =    6.304E+0010 [Tick^4]  Variance of Variances
MinsMSE =     1.918E+0001 [Tick^2]    Linear Regression MSE
R2 =          9.694E-0001 [-]         Determination Coeficient
Overhead =   100.6       [Tick]      Measurement Overhead
Measure =     0.9         [Tick]       Code Execution Time


Code (With loop and target) Select
Spurious =   2           [-]           Number of Spurious min. Values
MaxMaxDiff =  95388        [Tick]     Max. Spread of the Values
MeanVar =     3.608E+0005 [Tick^2]    Mean Variance
VarOfVars =    2.637E+0011 [Tick^4]  Variance of Variances
MinsMSE =     1.112E+0002 [Tick^2]    Linear Regression MSE
R2 =          9.994E-0001 [-]         Determination Coeficient
Overhead =   82.7         [Tick]       Measurement Overhead
Measure =     15.4       [Tick]      Code Execution Time


HSE
Equations in Assembly: SmplMath

Biterider

Hi HSE
I agree with you that the "quality" of the measurement in Windows is not as good as in the other environments mentioned.
The question we need to answer is whether it is good enough for code timing.

The criteria presented by Paolini and the method he showed seem to me to be much better than the simple and fallacious start-stop measurement method.
Repeatability and linearity are the criteria for me to validate the method. Variance, variance of variances and so on are parameters that define accuracy and dispersion.
The last parameter is much worse under Windows, as expected, but still the approach is reasonable and well applicable.

Just my 2 cents  :biggrin:

Biterider

HSE

#3
Hi Biterider!

Quote from: Biterider on January 13, 2023, 02:40:25 AM
The criteria presented by Paoloni and the method he showed seem to me to be much better than the simple and fallacious start-stop measurement method.

No doubt.

Quote from: Biterider on January 13, 2023, 02:40:25 AM
Repeatability and linearity are the criteria for me to validate the method.

:biggrin: Not for Paoloni. Repeatability is just an expression of measurement variance, and that is the thing. But results are very stable because are processed sample groups, not individual samples  :thumbsup:

Quote from: Biterider on January 13, 2023, 02:40:25 AM
Variance, variance of variances and so on are parameters that define accuracy and dispersion.

Exatly this is what make a method reliable or not for Paoloni.

Quote from: Biterider on January 13, 2023, 02:40:25 AM
Just my 2 cents  :biggrin:

:thumbsup: A complete piece.

I think a more complex design is necesary because I see little strange things, also in UEFI, perhaps related to core speed variations. Probably that is a processor response to load or temperature. But that is a complete speculation  :biggrin:
Equations in Assembly: SmplMath

Biterider

Hi
I took a closer look at the data from the benchmark application and found an issue that also occurs in other related projects.

The problem lies in the linear regression, which assumes that the first data is at X=0, but in reality it is at X=1.
For this reason, the offset, called overhead in this algorithm, needs to be corrected by one slope unit.

I have made the corrections in the Github projects, but those who use the algorithm must also fix their code.

Biterider



HSE

Hi Biterider!

Quote from: Biterider on January 20, 2023, 02:19:02 AM
The problem lies in the linear regression, which assumes that the first data is at X=0, but in reality it is at X=1.

No. You begin with x=0:    xor rsi, rsi
         ···
    xor edi, edi
        ···
    mov ebx, esi


Just in the slope there is an extra because:
    dec ebx

HSE
Equations in Assembly: SmplMath

Biterider

Hi HSE
No, it is like I said.
The index starts with zero but for the following calculations you have to think of it as X=1.
That is why you have to correct the offset/overhead value, otherwise you will get the offset at X=1 and that is not what you want.
The axis intersection is at X=0.


Biterider

HSE

Quote from: Biterider on January 20, 2023, 04:37:39 AM
Quote from: HSE on January 20, 2023, 04:18:26 AM
Quote from: Biterider on January 20, 2023, 02:19:02 AM
Hi
No..
No...

:dazzled:

What I can say? With same code, here begin with x=0  :biggrin::
SIZE_OF_STAT  equ 2
BOUND_OF_LOOP equ 3

ebx = 0 [BenchmarkWin.asm, 127]
ebx = 0 [BenchmarkWin.asm, 127]
ebx = 1 [BenchmarkWin.asm, 127]
    testing opcodes [BenchmarkWin.asm, 130]
ebx = 1 [BenchmarkWin.asm, 127]
    testing opcodes [BenchmarkWin.asm, 130]
ebx = 2 [BenchmarkWin.asm, 127]
    testing opcodes [BenchmarkWin.asm, 130]
    testing opcodes [BenchmarkWin.asm, 130]
ebx = 2 [BenchmarkWin.asm, 127]
    testing opcodes [BenchmarkWin.asm, 130]
    testing opcodes [BenchmarkWin.asm, 130]


Equations in Assembly: SmplMath

Biterider

Hi HSE
Sorry, I think I was wrong.  :sad:
The first iteration does not execute the test code and as such, it is evaluated at X=0.
I'll have to roll back the code.


Biterider

HSE

Equations in Assembly: SmplMath