ML.exe super-slow with large BYTE values?

[jj2007]

Glenfiddich, Dimple

You drunkards!!!! Read reply #8 carefully!!!

First, it's clearly addressed to Hutch
Second, it relates to a source here on the forum that deallocates RAM before the user hits Alt F4
Third, it only says I owe him a bottle of malt. In case he can present that source, and it was posted before today, he'll get a PM explaining where he can pick the malt (there are many nice places in Northern Italy ).

@Hutch: Yes, I'll also offer you a cappuccino :icon14:

[Gunther]

Jochen,

First, it's clearly addressed to Hutch

I'm so sorry.

Second, it relates to a source here on the forum that deallocates RAM before the user hits Alt F4

That's a bit quibbling.

(there are many nice places in Northern Italy ).

Of course; I would say near the  Lago di Garda.

Gunther

[jj2007]

Second, it relates to a source here on the forum that deallocates RAM before the user hits Alt F4

That's a bit quibbling.

Well, no, it's really the essence of the argument:

Dynamic allocation has the advantage that it only occupies memory when its needed by an app, not for the life of the app

This is the whole argument in favour of dynamic allocation: That you can allocate and release memory as needed, in reacting to user requests, e.g. for opening a new browser tab. All examples posted above as "evidence" don't do that - they allocate memory for the life of the app. And that can be done more efficiently via .data? ...

[dedndave]

most programs in here - we don't have to use Alt-F4   

[jj2007]

most programs in here - we don't have to use Alt-F4   

Yes, they end without user intervention

[hutch--]

 

I won't hold you to the bottle of Glenfiddich as I have one open and a new one as a spare that I got for my last birthday.  :P

[RuiLoureiro]

Dynamic allocation has the advantage that it only occupies memory when its needed by an app

Hutch,
If you can show me one source here on the forum that allocates some megabytes and deallocates them before the user hits Alt F4 (that is the meaning of "temporarily"), then I owe you a bottle of malt :icon14:

Hummmm,
                  i have apps that do this, but it deallocates only when we close the app ! I never do that before. So, Jochen, save the bottle or ... share it with me  ;) :P
:icon14:

[dedndave]

i used to drink a little scotch - lol

nowdays, i like to have one of these around - which lasts me 2 or 3 years
by the time i get to the end, it's 12 years old   :lol:

good ole Kentucky "sippin" whiskey   



just like grandad used to make

[Gunther]

Dave,

it's Bourbon; we're talking about Whisky. There is a mile of difference.

Gunther

[KeepingRealBusy]

Second, it relates to a source here on the forum that deallocates RAM before the user hits Alt F4

That's a bit quibbling.

Well, no, it's really the essence of the argument:

Dynamic allocation has the advantage that it only occupies memory when its needed by an app, not for the life of the app

This is the whole argument in favour of dynamic allocation: That you can allocate and release memory as needed, in reacting to user requests, e.g. for opening a new browser tab. All examples posted above as "evidence" don't do that - they allocate memory for the life of the app. And that can be done more efficiently via .data? ...

JJ,

I don't see the problem with keeping the memory around until the end of the app. It is only virtual memory. If I am not using it, then the system will page it out if it needs some real memory. And this paging also happens even if I allocate 3.5 GB of memory and am currently using all of it since the system only allows a limited number of pages that are currently paged in - the least currently accessed page will page out and the new page I need RIGHT NOW will page in, only to be paged out at some later time.

This paging problem requires careful program design to keep accesses within certain pages (whenever possible) instead of randomly accessing a BYTE st some low address, and then a BYTE at a very high address, then in the middle.

In some cases it is faster to read in a huge buffer then process the buffer multiple times, selecting the records for one output file and skipping the others, then re-processing the buffer and selecting the second file records for output. Actually you can test by timing so see how many output buffers you can select in one pass without encountering severe paging. In my case I am reading a file into an input buffer and splitting it into 512 output buffers (no writing required) where the input is 3.7 GB, then I access each output buffer in turn and split the content into the input buffer, but split into 4 pieces, then finally write out the 4 pieces (with an index header) to the actual output file.

This worked for the first half of the data. These initial data files had all  the same sizes, and the output files were all the same sizes. The second half of the data files, however, is not so evenly dispersed. The files are from 5.080 GB to 2.547 GB. The biggest files when split to 512 buffers will not fit in the memory buffers, so I am left with several options as to how to proceed. I cannot split the data into 256 buffers, then later split the data into 8 pieces because the data will still overflow the buffers - there will be twice as much data in half of the buffer count. I cannot even read the entire file into memory and split off (process) only one output file at a time because I can only allocate 3.5 GB of memory and need to split 5.080GB of data, so I am left with writing intermediate files.  This doubles the file I/O time which is the largest part of the process (6 hrs writing  and 1.5 hr processing on the initial data files).

Still thinking about what to do with this.

Any one have any suggestions?

Dave.

[hutch--]

Dave,

That is what memory mapped files are for. You could tile it with disk IO if the data can be processed it blocks but a memory mapped file is probably the better way.

[KeepingRealBusy]

Steve,

Thank you for the hint. I have never used memory mapped I/O. Any good links to read?

Dave.

[hutch--]

Dave,

There is an example in the masm32 examples called mmfdemo.asm but its more aimed at sharing the memory between apps than with a disk file. The API are CreateFileMapping(), MapViewOfFile(), UnmapViewOfFile() and CloseHandle().

I don't immediately have a disk file version but the WIN32 HLP looks pretty clear to use.

[KeepingRealBusy]

Steve,

Thanks for the info. I got into some of the msdn info on line, still studying. Really looking for a compelling reason to switch to this rather than OpenFile, SeekFile, ReadFile, WriteFile etc.

Dave.

[jj2007]

I don't see the problem with keeping the memory around until the end of the app. It is only virtual memory. If I am not using it, then the system will page it out if it needs some real memory.

Dave,

Yes, I agree fully - and that's the whole point of the whisky controversy: If the memory is needed for the lifetime of the application, then it doesn't matter if you use dynamic allocation or the .data? section, the OS couldn't care less.

But you are introducing a new aspect: paging. One could argue that dynamic memory is better because the OS can park it in the pagefile if not needed, while .data? mem is "worse" because it blocks other applications. But actually, that's not the case: even uninitialised data gets paged in only when needed, as the attached little app demonstrates. So it is really difficult to find a factual argument against using uninitialised data, except for the one case that is disputed above: That you want to deallocate it without exiting the application.

P.S.: bufsize=1024*1024*1024 works fine, too. The app starts half a second slower, though

include \masm32\MasmBasic\MasmBasic.inc        ; download
        Init
        bufsize=1024*1024*32   ; we want 32 Megabytes of unitialised memory
        crtbuf fatbuffer, bufsize, 16
        mov edi, fatbuffer
        invoke MbCopy, edi, Chr$("Open Task Manager, look at mem usage, Alt Tab back here and hit any key"), -1
        Inkey edi
        xor ecx, ecx
        .Repeat
                mov byte ptr [edi+ecx], 0
                add ecx, 127
        .Until ecx>=bufsize
        inkey "more?"
        Exit
end start