ML.exe super-slow with large BYTE values?

[MtheK]

  If I code this and assemble it, it runs in about a half-second:

DSNBUFFERL DWORD 32767
DSNBUFFER1 BYTE 32767 DUP(00h)
DSNBUFFER2 BYTE 32767 DUP(00h)

time:  10:34:46.13 to 10:34:46.62


  Yet, if I do the same program and change the above to this:

DSNBUFFERL DWORD 1024000               
DSNBUFFER1 BYTE 1024000 DUP(00h)
DSNBUFFER2 BYTE 1024000 DUP(00h)

it runs about 455x slower, about 4 minutes:

time:  10:37:38.67 to 10:41:21.90

  My program runs the same, albeit a bit faster.

  Is this expected?

[jj2007]

Is this expected?

Yes, it's a known bug. Workarounds:
- use JWasm
- use the MasmBasic crtbuf macro, or something similar:

crtbuf DSNBUFFER1, 1024000  ; in the .code section

crtbuf MACRO var, BufLen, msAlign:=<4>   ; cb pBuffer, 1000 [, 16]
LOCAL cblabel
.data?
align msAlign
  cblabel LABEL BYTE
  var equ offset cblabel
  ORG $+BufLen-1
  db ?
.code
ENDM

[dedndave]

yes - it is a known masm bug

there are ways to avoid it
a simple way is to break up the define
i think if you break the 1024000 byte buffers into 16 64000 byte buffers, it will build faster

Code Select Expand

DSNBUFFER1 BYTE 64000 DUP(00h)
           BYTE 64000 DUP(00h)
           BYTE 64000 DUP(00h)

;etc

the ORG macro that Jochen mentioned is also a good way

of course the SIZEOF and LENGTHOF operators won't work
but, you can put a label at the end and subtract addresses to get past that

[hutch--]

The general drift is to use allocated memory from the OS, not the initialised data section when you need more than trivial amounts. Its generally classed as bad practice to do this as it blows the size of the file out by the amount of memory you configure this way. With allocated memory you can routinely allocate up to near 2 gigabytes if you have enough memory on the machine.

[Rockoon]

While "bad practice", you dont have to deal with a runtime out-of-memory error if you never call malloc() or its equivalents.

[dedndave]

something i forgot to mention...
if you put it in the uninitialized data section, it will be filled with 0's
and the EXE will be a lot smaller   

[jj2007]

if you put it in the uninitialized data section, it will be filled with 0's
and the EXE will be a lot smaller   

In fact that's what the crtbug macro does. Full example:

include \masm32\include\masm32rt.inc

crtbuf MACRO var, BufLen, msAlign:=<4>
LOCAL cblabel
.data?
align msAlign
  cblabel LABEL BYTE
  var equ offset cblabel
  ORG $+BufLen-1
  db ?
.code
ENDM

.code
start:   crtbuf mypath, MAX_PATH      ; simple version
   crtbuf MySSE2buffer, 1024*1024, 16      ; with optional argument specifying alignment
   invoke lstrcpy, offset mypath, rv(GetCommandLine)
   MsgBox 0, offset mypath, "The commandline:", MB_OK
   exit
end start

Re "bad practice": As Rockoon already wrote, using .data? avoids calling the alloc family. In practice, the OS couldn't care less if it's .data or somewhere else on the heap; the only occasion where alloc (HeapAlloc, GlobalAlloc, VirtualAlloc, ...) shows an advantage is when you need temporarily a dozen megabytes or so (a case which is rare in this forum but certainly relevant for Firefox developers).

[hutch--]

 

Since when was a few megabytes "rare" in MASM coding. I deviate only when I need a gigabyte or so.  :P  Dynamic allocation has the advantage that it only occupies memory when its needed by an app, not for the life of the app although it can be done that way dynamically as well.

[jj2007]

Dynamic allocation has the advantage that it only occupies memory when its needed by an app

Hutch,
If you can show me one source here on the forum that allocates some megabytes and deallocates them before the user hits Alt F4 (that is the meaning of "temporarily"), then I owe you a bottle of malt :icon14:

[GoneFishing]

maybe http://masm32.com/board/index.php?topic=2162.0  reply#8  ? 

[dedndave]

careful how you word that, Jochen - the postage could get expensive   :redface:

[Gunther]

Jochen,

that DPMI application from the 16 bit sub-forum allocates temporary all available RAM and frees it before it terminates. That's on my machine approximately 3.5 GB. Is that enough? By the way, I would prefer Glemfiddich.

Gunther

[jj2007]

maybe http://masm32.com/board/index.php?topic=2162.0  reply#8  ? 

I don't feel it fits into the description of a normal app that temporarily allocates a few megabytes...

My memory allocation always allocates the largest block possible (VirtualAlloc), and does this 10 times, then de-allocates a block thought to be large enough to bring in the debugger then does an int 3. If the program dies with no messages (instant death), then the reserved block is not large enough, double the size ...

that DPMI application from the 16 bit sub-forum allocates temporary all available RAM and frees it before it terminates.

The program checks the available extended memory, allocates that amount of memory, fills the memory with values and reads the values back for printing at the text screen. That's all.

Again, this is not an app that temporarily allocates memory for performing a specific task. It allocates memory for the whole life of the application, and that could be done as well in .data. Since it's a DOS app, the user does not hit Alt F4, I guess, but anyway, the deallocation happens after the user decides to terminate the application, right? Or did I misunderstand something, and the user can continue to run the app without the "temporarily" allocated memory??

There is a reason why I mentioned Firefox. A browser opens a tab and needs some megabytes for that. The user closes the tab, and Firefox gently releases the memory (because otherwise it would be bashed in various benchmarking sites as a memory hog). That is temporary allocation, because deallocation occurs despite the fact that the app keeps running.

[japheth]

jwasm also very often allocates a few megabytes temporarily. The size depends on the complexity of the assembly source it has to digest, of course, but 4 MB is usually exceeded for a Windows applications.

I want Glenfiddich, at least 24 years old!

[nidud]

deleted