Benchmark (for assemblers and compilers)

[jj2007]

IMHO it makes little sense to benchmark one single feature like "calculate PI" or "multiply matrices". A reasonable testbed should contain a range of common tasks like

arithmetic calculations
parsing
converting
sorting (strings, int, REALx)
tight loops
string management (reading, writing, appending, concat)
arrays (creation, inserting and deleting elements, sorting)
etc.

Below a test case, using this free 20 MB database, a csv file. The snippet measures the time needed to load the database and calculate the World average for the indicator "Population undernourished, percentage" in the year 2000. 234 of 44k lines contain this indicator, the values are in the "2000" column. M$ Excel loads the file in about 3 seconds. Easy enough, no?

include \masm32\MasmBasic\MasmBasic.inc
  SetGlobals column, rowMatch, valMatch, double tempVal, sum
  Init
  NanoTimer()      ; start timing
  Recall "MdgData.csv", L$(), csv      ; UN database available here
  .Repeat
       .Break .if !StringsDiffer(L$(0, column), "2000")  ; find the year 2000 column
       inc column
  .Until column>99
  For_ ct=0 To L$(?)-1
        .if Instr_(L$(ct), "Population undernourished, percentage")  ; check if row contains the string
               MovVal tempVal, L$(ct, column)  ; load the value to a temporary variable
               .if signed edx>0        ; check if the cell was valid
                               inc valMatch    ; count valid matches
                               ; Print Str$("line %___i", ct), Str$(" %3f\n", tempVal)   ; for testing
                               fld tempVal     ; tmp on FPU
                               fadd sum        ; add current sum
                               fstp sum        ; save as new sum
               .endif
                inc rowMatch
        .endif
  Next
  Inkey Cpu$(), CrLf$, NanoTimer$(), Str$(" for loading the file and processing %i matches", rowMatch), Str$(" with an average of %5f", sum/valMatch)
EndOfCode

Output:

Code Select Expand

Intel(R) Core(TM) i5-2450M CPU @ 2.50GHz
18 ms for loading the file and processing 234 matches with an average of 15.996

[aw27]

IMHO it makes little sense to benchmark one single feature like "calculate PI" or "multiply matrices". A reasonable testbed should contain a range of common tasks like

arithmetic calculations
parsing
converting
sorting (strings, int, REALx)
tight loops
string management (reading, writing, appending, concat)
arrays (creation, inserting and deleting elements, sorting)
etc.

Below a test case, using this free 20 MB database, a csv file. The snippet measures the time needed to load the database and calculate the World average for the indicator "Population undernourished, percentage" in the year 2000. 234 of 44k lines contain this indicator, the values are in the "2000" column. M$ Excel loads the file in about 3 seconds. Easy enough, no?

include \masm32\MasmBasic\MasmBasic.inc
  SetGlobals column, rowMatch, valMatch, double tempVal, sum
  Init
  NanoTimer()      ; start timing
  Recall "MdgData.csv", L$(), csv      ; UN database available here
  .Repeat
       .Break .if !StringsDiffer(L$(0, column), "2000")  ; find the year 2000 column
       inc column
  .Until column>99
  For_ ct=0 To L$(?)-1
        .if Instr_(L$(ct), "Population undernourished, percentage")  ; check if row contains the string
               MovVal tempVal, L$(ct, column)  ; load the value to a temporary variable
               .if signed edx>0        ; check if the cell was valid
                               inc valMatch    ; count valid matches
                               ; Print Str$("line %___i", ct), Str$(" %3f\n", tempVal)   ; for testing
                               fld tempVal     ; tmp on FPU
                               fadd sum        ; add current sum
                               fstp sum        ; save as new sum
               .endif
                inc rowMatch
        .endif
  Next
  Inkey Cpu$(), CrLf$, NanoTimer$(), Str$(" for loading the file and processing %i matches", rowMatch), Str$(" with an average of %5f", sum/valMatch)
EndOfCode

Output:

Code Select Expand

Intel(R) Core(TM) i5-2450M CPU @ 2.50GHz
18 ms for loading the file and processing 234 matches with an average of 15.996

Is this a test for hard disk performance?

[felipe]

microprocessors are so complex beast, but all the circuitry around them are complex too (with more microprocessors running around too). so there can be a lot of differents test for speed: with gpu, i/o devices, network card (nic), and more...
even if you approach a super complete test including all this computer components, you will not able (most probably) to execute with success this same test in a different computer, because most probably, there will be different components or even specific different time situations... :idea:

but still may be a lot of fun!  :icon14:

[TimoVJL]

Odd csv-file, maybe a csv generator problem?

Code Select Expand

"3611","Pew Internet Project, "Internet, Broadband and cell phone statistics," January 5, 2010, http://www.pewinternet.org/~/media//Files/Reports/2010/PIP_December09_update.pdf."
Project, "Internet and statistics," January

[jj2007]

Is this a test for hard disk performance?

When you run it the first time, yes. Afterwards the 20MB are in the file cache. But you are right, the timings should be split (source & exe attached):

Code Select Expand

Intel(R) Core(TM) i5-2450M CPU @ 2.50GHz
12 ms for reading the file into a buffer
13 ms for converting the buffer content to a string array
3147 µs for processing 234 matches with an average of 15.996

@Timo: Yes, there are some cute real world problems in that file. Row 44353 should read as follows:

3592<TAB>Pew Internet Project, "Internet, Broadband and cell phone statistics," January 5, 2010, http://www.pewinternet.org/~/media//Files/Reports/2010/PIP_December09_update.pdf.

Drag the csv over the attached exe (work in progress, undo won't work properly!), then hit Ctrl F and search for Pew Internet. Or, even better, for Where Have All The Poor Gone, which gets incorrectly decoded by M$ Excel as WB Staff estimates. Cambodia Poverty Assessment 2013 Where Have All The Poor Gone?" (trailing quote but none on entry). That one is indeed a bit tricky, here is how it looks when opened with qEditor:

Code Select Expand

"1575","WB Staff estimates. Cambodia Poverty Assessment 2013 "Where Have All The Poor Gone?",  May 2013. Cambodia Socio-Economic Survey 2009."

[TimoVJL]

Sadly average value differ in my test :(

Code Select Expand

Rows: 43654 Cols: 248 152ms
234 11.512821is use MDG_Export_20190317_120610383.csv

EDIT: Benchmark result:

Code Select Expand

AMD Athlon(tm) II X2 220 Processor
25 ms for reading the file into a buffer
25 ms for converting the buffer content to a string array
5507 µs for processing 234 matches with an average of 15.754

[jj2007]

Sadly average value differ in my test

How many valid matches?

Inkey NanoTimer$(), Str$(" for processing %i matches", rowMatch), Str$(" (of which %i valid)", valMatch), Str$(" with an average of %5f", sum/valMatch)

Code Select Expand

3852 µs for processing 234 matches (of which 171 valid) with an average of 15.996

I use MovVal, which returns the number of usable digits in edx; therefore the .if signed edx>0 in the source. For Excel, the average is 15.9959. If you multiply your value with 234/171, your result is quite close (15.7544), the reason being that most if not all of the invalid matches are empty. But you are still a little bit off then. Your source is in C, I suppose?

[TimoVJL]

Code Select Expand

Rows: 43654 Cols: 248 145ms
234 matches 171 values, average 15.754
EDIT: fixed

Code Select Expand

Rows: 43654 Cols: 248 135ms
234 matches 171 values, average 15.996just problem with ',' conversion to '.'

EDIT: msvc 19.16 versions using msvcrt.dll

EDIT: a test package of gcc 8 msvc 19.16 Pellesc 9

Code Select Expand

DSVRead_Test1msvc.exe
C compiler: Microsoft Visual C/C++ Version: 1916 Toolset: 191627026
DSVRead: 110ms
Rows: 43654 Cols: 248
123ms
234 matches 171 values, average 15.996

DSVRead_Test1msvcX64.exe
C compiler: Microsoft Visual C/C++ Version: 1916 Toolset: 191627026
DSVRead: 103ms
Rows: 43654 Cols: 248
116ms
234 matches 171 values, average 15.996

DSVRead_Test1gcc8X64.exe
C compiler: MingW64 6.0
DSVRead: 111ms
Rows: 43654 Cols: 248
126ms
234 matches 171 values, average 15.996

DSVRead_Test1.exe
C compiler: Pelles, Version 900
DSVRead: 115ms
Rows: 43654 Cols: 248
129ms
234 matches 171 values, average 15.996

DSVRead_Test164.exe
C compiler: Pelles, Version 900
DSVRead: 111ms
Rows: 43654 Cols: 248
126ms
234 matches 171 values, average 15.996


[jj2007]

Pretty close but no cigar yet. Can you print the valid numbers, together with their row number? Here are mine:

Code Select Expand

line   30 45.2
line  204 5.00
line  378 8.70
line  726 5.00
line  900 51.1
line 1422 5.00
line 1596 21.4
line 1944 5.00
line 2118 5.00
line 2292 22.5
line 2814 23.1
line 2988 5.00
line 3162 5.00
line 3336 5.00
line 3510 6.50
line 3684 23.9
line 3858 5.00
line 4206 34.6
line 4380 5.00
line 4554 35.6
line 4728 12.3
line 5076 5.00
line 5250 5.00
line 5424 26.6
line 5772 32.0
line 5946 32.3
line 6120 5.00
line 6294 18.7
line 6642 44.1
line 6816 40.1
line 7164 5.00
line 7338 16.2
line 7860 9.90
line 8208 35.9
line 8556 5.20
line 8730 14.9
line 8904 5.00
line 9078 5.60
line 9252 5.00
line 9426 5.00
line 9948 5.00
line 10122 52.4
line 10470 30.7
line 10644 17.8
line 10818 5.00
line 10992 12.5
line 11514 5.00
line 11688 57.9
line 12036 5.00
line 12384 5.00
line 12558 5.00
line 12732 5.00
line 13254 5.00
line 13428 14.1
line 13602 14.8
line 13776 5.00
line 13950 17.5
line 14124 5.00
line 14298 5.00
line 14472 5.00
line 15168 22.1
line 15342 27.2
line 15516 28.4
line 15690 10.4
line 15864 55.2
line 16038 19.0
line 16212 5.00
line 16386 5.00
line 16560 17.0
line 16734 17.2
line 16908 5.20
line 17082 24.6
line 17256 5.00
line 17430 5.00
line 17604 5.00
line 17778 7.80
line 17952 5.00
line 18126 7.00
line 18300 5.00
line 18474 32.3
line 18648 5.00
line 18822 37.9
line 18996 5.00
line 19170 5.00
line 19344 15.2
line 19518 39.2
line 19692 5.00
line 19866 5.00
line 20040 13.0
line 20214 36.5
line 20562 5.00
line 20736 5.00
line 20910 5.00
line 21084 34.8
line 21258 28.6
line 21432 5.00
line 21606 11.8
line 21780 13.9
line 21954 5.00
line 22476 11.5
line 22650 7.10
line 22998 5.00
line 23346 5.00
line 23520 38.2
line 23694 5.00
line 24042 6.50
line 24216 42.0
line 24390 52.4
line 24564 30.4
line 24912 22.2
line 25086 5.00
line 25608 5.00
line 25782 34.8
line 25956 22.8
line 26130 9.20
line 26652 5.00
line 26826 10.4
line 27000 22.4
line 27348 27.4
line 27696 13.3
line 27870 21.6
line 28044 21.3
line 28218 5.00
line 28392 5.00
line 28914 5.00
line 29262 5.00
line 29436 5.00
line 29610 60.6
line 30306 5.00
line 30480 18.9
line 30654 6.60
line 30828 5.00
line 31002 21.5
line 31176 5.00
line 31350 29.4
line 31524 5.00
line 32046 38.0
line 32394 5.00
line 32568 5.00
line 32742 15.0
line 33090 5.00
line 33438 5.00
line 33612 29.9
line 34134 24.1
line 34308 14.1
line 34482 21.7
line 34656 5.00
line 34830 5.00
line 35178 38.8
line 35352 19.0
line 35526 5.00
line 35700 43.9
line 35874 29.2
line 36396 13.0
line 36570 5.00
line 36744 5.00
line 36918 9.00
line 37440 28.4
line 37614 5.00
line 37962 5.00
line 38136 5.00
line 38310 36.8
line 38484 5.00
line 38832 5.00
line 39006 11.5
line 39180 8.10
line 39354 16.6
line 39528 28.1
line 40050 29.6
line 40398 42.9
line 40572 43.7
187 ms for processing 234 matches (of which 171 valid) with an average of 15.996

[jj2007]

EDIT: a test package of gcc 8 msvc 19.16 Pellesc 9

Very nice, thanks :t

PellesC is pretty fast in these tests:

Code Select Expand

DSVRead_Test1msvc.exe
C compiler: Microsoft Visual C/C++ Version: 1916 Toolset: 191627026
DSVRead: 68ms
Rows: 43666 Cols: 196
78ms
234 matches 171 values, average 15.996

DSVRead_Test1msvcX64.exe
C compiler: Microsoft Visual C/C++ Version: 1916 Toolset: 191627026
DSVRead: 63ms
Rows: 43666 Cols: 196
73ms
234 matches 171 values, average 15.996

DSVRead_Test1gcc8X64.exe
C compiler: MingW64 6.0
DSVRead: 74ms
Rows: 43666 Cols: 196
86ms
234 matches 171 values, average 15.996

DSVRead_Test1.exe
C compiler: Pelles, Version 900
DSVRead: 66ms
Rows: 43666 Cols: 196
76ms
234 matches 171 values, average 15.996

DSVRead_Test164.exe
C compiler: Pelles, Version 900
DSVRead: 69ms
Rows: 43666 Cols: 196
80ms
234 matches 171 values, average 15.996

[TimoVJL]

Now pcc32:

Code Select Expand

DSVRead_Test1pcc32.exe
C compiler: Microsoft Visual C/C++ Version: 600
DSVRead: 129ms
Rows: 43654 Cols: 248
143ms
234 matches 171 values, average 15.996

[jj2007]

Surprisingly slow, PellesC beats it hands down:

Code Select Expand

DSVRead_Test1msvc.exe
C compiler: Microsoft Visual C/C++ Version: 1916 Toolset: 191627026
DSVRead: 69ms
Rows: 43666 Cols: 196
76ms
234 matches 171 values, average 15.996

DSVRead_Test1msvcX64.exe
C compiler: Microsoft Visual C/C++ Version: 1916 Toolset: 191627026
DSVRead: 66ms
Rows: 43666 Cols: 196
72ms
234 matches 171 values, average 15.996

DSVRead_Test1gcc8X64.exe
C compiler: MingW64 6.0
DSVRead: 76ms
Rows: 43666 Cols: 196
86ms
234 matches 171 values, average 15.996

DSVRead_Test1.exe
C compiler: Pelles, Version 900
DSVRead: 66ms
Rows: 43666 Cols: 196
73ms
234 matches 171 values, average 15.996

DSVRead_Test164.exe
C compiler: Pelles, Version 900
DSVRead: 66ms
Rows: 43666 Cols: 196
73ms
234 matches 171 values, average 15.996

C compiler: Microsoft Visual C/C++ Version: 600
DSVRead: 79ms
Rows: 43666 Cols: 196
87ms
234 matches 171 values, average 15.996

[TimoVJL]

As slow as gcc 8 sample  :P

[jj2007]

There is a very big test here, where Gcc8 comes out very well. What surprises me is the good result for Pelles C. Once it was considered too slow to be included in benchmark exercises such as https://www.willus.com/ccomp_benchmark2.shtml?p2

[TimoVJL]

@jj2007 gcc 8.2 don't like your CPU :icon_confused:
In Windows gcc use msvcrt.dll, but it's own __strtod().

Maybe i did something wrong, so here are those test source-files.
Define NO_MALLOC if you don't want  to use malloc().

Csv-file used for testing was download from here

gcc 8.2 used in test, can downloaded from here

EDIT: that pcc32 case, is it really worth to use a bloated compiler for simpler tasks?
and only about of 25% faster in simple tasks.

Right tools for tasks:
  if you want a bloated program , paid by size, use Delphi/Kylix
  if you want a slow prorgam, paid by slowness, choose Java or C# or VB.Net
otherwise just use an assembler or C