Hi
I spent some time adapting the code from this thread http://masm32.com/board/index.php?topic=10099.msg110903#msg110903 (http://masm32.com/board/index.php?topic=10099.msg110903#msg110903) to run on Windows.
The code was originally developed for Linux and adapted for UEFI and now further developed for Windows.
The intent was to see if such a benchmark was feasible and gave reproducible results.
Looking at the numbers and comparing them to the UEFI version, the code execution time looks very similar and the coefficient of determination R2 looks remarkably good. :cool:
If you want to test it, the code is on Github and the binary in the attachment. It requires DebugCenter to get the results.
Biterider
Hi Biterider!!
:thumbsup: Fantastic! Very simple to test little algorithms.
Spurious = 16 [-] Number of Spurious min. Values
MaxMaxDiff = 84868 [Tick] Max. Spread of the Values
MeanVar = 4.166E+0005 [Tick^2] Mean Variance
VarOfVars = 1.214E+0012 [Tick^4] Variance of Variances
MinsMSE = 5.546E+0002 [Tick^2] Linear Regression MSE
R2 = 9.973E-0001 [-] Determination Coeficient
Overhead = 78.4 [Tick] Measurement Overhead
Measure = 15.5 [Tick] Code Execution TimeQuote from: Biterider on January 12, 2023, 05:39:00 AM
the coefficient of determination R2 looks remarkably good. :cool:
You can't have a bad R
2, forget that :biggrin:. You have to see variance, and Variance of Variances is at list 10
6 times bigger than microLinux or UEFI.
Don't look very good at all:
Spurious = 9 [-] Number of Spurious min. Values
MaxMaxDiff = 89794 [Tick] Max. Spread of the Values
MeanVar = 1.005E+0005 [Tick^2] Mean Variance
VarOfVars = 4.369E+0010 [Tick^4] Variance of Variances
MinsMSE = 2.145E+0000 [Tick^2] Linear Regression MSE
R2 = 8.739E-0003 [-] Determination Coeficient
Overhead = 101.4 [Tick] Measurement Overhead
Measure = 0.0 [Tick] Code Execution Time
Spurious = 1 [-] Number of Spurious min. Values
MaxMaxDiff = 88474 [Tick] Max. Spread of the Values
MeanVar = 1.095E+0005 [Tick^2] Mean Variance
VarOfVars = 6.304E+0010 [Tick^4] Variance of Variances
MinsMSE = 1.918E+0001 [Tick^2] Linear Regression MSE
R2 = 9.694E-0001 [-] Determination Coeficient
Overhead = 100.6 [Tick] Measurement Overhead
Measure = 0.9 [Tick] Code Execution Time
Spurious = 2 [-] Number of Spurious min. Values
MaxMaxDiff = 95388 [Tick] Max. Spread of the Values
MeanVar = 3.608E+0005 [Tick^2] Mean Variance
VarOfVars = 2.637E+0011 [Tick^4] Variance of Variances
MinsMSE = 1.112E+0002 [Tick^2] Linear Regression MSE
R2 = 9.994E-0001 [-] Determination Coeficient
Overhead = 82.7 [Tick] Measurement Overhead
Measure = 15.4 [Tick] Code Execution Time
HSE
Hi HSE
I agree with you that the "quality" of the measurement in Windows is not as good as in the other environments mentioned.
The question we need to answer is whether it is good enough for code timing.
The criteria presented by Paolini and the method he showed seem to me to be much better than the simple and fallacious start-stop measurement method.
Repeatability and linearity are the criteria for me to validate the method. Variance, variance of variances and so on are parameters that define accuracy and dispersion.
The last parameter is much worse under Windows, as expected, but still the approach is reasonable and well applicable.
Just my 2 cents :biggrin:
Biterider
Hi Biterider!
Quote from: Biterider on January 13, 2023, 02:40:25 AM
The criteria presented by Paoloni and the method he showed seem to me to be much better than the simple and fallacious start-stop measurement method.
No doubt.
Quote from: Biterider on January 13, 2023, 02:40:25 AM
Repeatability and linearity are the criteria for me to validate the method.
:biggrin: Not for Paoloni. Repeatability is just an expression of measurement variance, and that is the thing. But results are very stable because are processed sample groups, not individual samples :thumbsup:
Quote from: Biterider on January 13, 2023, 02:40:25 AM
Variance, variance of variances and so on are parameters that define accuracy and dispersion.
Exatly this is what make a method reliable or not for Paoloni.
Quote from: Biterider on January 13, 2023, 02:40:25 AM
Just my 2 cents :biggrin:
:thumbsup: A complete piece.
I think a more complex design is necesary because I see little strange things, also in UEFI, perhaps related to core speed variations. Probably that is a processor response to load or temperature. But that is a complete speculation :biggrin:
Hi
I took a closer look at the data from the benchmark application and found an issue that also occurs in other related projects.
The problem lies in the linear regression, which assumes that the first data is at X=0, but in reality it is at X=1.
For this reason, the offset, called overhead in this algorithm, needs to be corrected by one slope unit.
I have made the corrections in the Github projects, but those who use the algorithm must also fix their code.
Biterider
Hi Biterider!
Quote from: Biterider on January 20, 2023, 02:19:02 AM
The problem lies in the linear regression, which assumes that the first data is at X=0, but in reality it is at X=1.
No. You begin with x=0:
xor rsi, rsi
···
xor edi, edi
···
mov ebx, esi
Just in the slope there is an extra because:
dec ebxHSE
Hi HSE
No, it is like I said.
The index starts with zero but for the following calculations you have to think of it as X=1.
That is why you have to correct the offset/overhead value, otherwise you will get the offset at X=1 and that is not what you want.
The axis intersection is at X=0.
Biterider
Quote from: Biterider on January 20, 2023, 04:37:39 AM
Quote from: HSE on January 20, 2023, 04:18:26 AM
Quote from: Biterider on January 20, 2023, 02:19:02 AM
Hi
No..
No...
:dazzled:
What I can say? With same code, here begin with x=0 :biggrin::
SIZE_OF_STAT equ 2
BOUND_OF_LOOP equ 3
ebx = 0 [BenchmarkWin.asm, 127]
ebx = 0 [BenchmarkWin.asm, 127]
ebx = 1 [BenchmarkWin.asm, 127]
testing opcodes [BenchmarkWin.asm, 130]
ebx = 1 [BenchmarkWin.asm, 127]
testing opcodes [BenchmarkWin.asm, 130]
ebx = 2 [BenchmarkWin.asm, 127]
testing opcodes [BenchmarkWin.asm, 130]
testing opcodes [BenchmarkWin.asm, 130]
ebx = 2 [BenchmarkWin.asm, 127]
testing opcodes [BenchmarkWin.asm, 130]
testing opcodes [BenchmarkWin.asm, 130]
Hi HSE
Sorry, I think I was wrong. :sad:
The first iteration does not execute the test code and as such, it is evaluated at X=0.
I'll have to roll back the code.
Biterider
:thumbsup: