Large buffers

[redskull]

In deed - Chunking makes it a bit faster - mseconds ?

As was said, a lot depends on how you are using it, but allocating a huge buffer and filling it is usually the worst possible way.  Hard drives read by sectors (512) and windows organizes them by cluster (4096), so if you do a 5MB blocking read, your program is going to grind to a halt while the drive spins around, transfers each sector, and then searches around for the next cluster, and does this over and over until the buffer is full.

If you are accessing the data in a sequential manner (start with byte 1, and process until the end), then all this searching time is wasted.  By establishing two buffers of the cluster size, you start processing the data after the first cluster is read in, while windows goes about filling the next one in the background.  In a perfect world you never stall for data except for the first read, and you essentially eliminate all your loading time entirely.

In the other usage pattern, you need to access any given byte of the file at any given time, so there is no way around having the whole thing loaded.  But by memory-mapping the file, windows doesn't load the appropriate 4K chunk until you try and access it, which at the very least will improve responsiveness, since the user doesn't have to sit through an awful loading screen while you read up every byte off the disk.  Or, even better, since the only chunks that get loaded are the ones that get read, if a particular use case only touches one cluster, you never end up loading the other twelve hundred at all.  Plus you free up that memory for your program to use in its working set, since windows no longer has to worry about paging out the buffer.

Plus, as an aside, this probably shouldn't be a static allocation anyway, since it seems far fetched that the only kind of file your program will need to process is one that's always going to be exactly 5 million bytes.  You are either wasting space if the file is smaller or missing functionality if the file is bigger (or, if you are in control of the external files and are going to ensure they are always said size, you are imposing an unnecessary limitation on yourself).

-r

[jj2007]

if you do a 5MB blocking read, your program is going to grind to a halt while the drive spins around, transfers each sector, and then searches around for the next cluster, and does this over and over until the buffer is full.

If you are accessing the data in a sequential manner (start with byte 1, and process until the end), then all this searching time is wasted.  By establishing two buffers of the cluster size, you start processing the data after the first cluster is read in,

Red,

Your theory is OK, and I really feel we should make it a nice timing project in the lab. Problems I see:

- disk cache *): after the first test, your 5MB will come in rather quickly (and is an uncached read realistic?)
- the handling of overlaps between the two buffers., i.e. checking for each byte if you need to switch the buffer needs time
- if your processing is not read-only, you'll need a copy of the two buffers
- re memory-mapped files: fine, could eliminate the need to check for overlaps, but what if you need to write into your buffer but not into your file? My Recall(), for example, does that; same for the line tokeniser recently made by Hutch.

Again: to be tested

*) see http://stackoverflow.com/questions/7405868/how-to-invalidate-the-file-system-cache/8113490#8113490 for a method that might flush the disk cache maintained by the OS; it won't affect the hard drive's cache, though - and that is the one that smoothens the physical access problems.

[dedndave]

i think it would be very difficult to get meaningful results
every machine/OS/setup/etc will yield different results - lol

[K_F]

With MASM you put it into a special section - .data? (uninitialized data). This section doesn't actually exist in the EXE image.

To think I've missed this, all these years...
So Sinsi.. you're saying that the .data? section is not in the compiled exe file.
Where would it be kept..
Would it be recorded in the PE header then, and space allocated when the exe is loaded ?
:icon_exclaim:

[dedndave]

of course it doesn't exist - at least, not byte-by-byte
there is a table entry that gives a size and offset - that's all that's needed (and a name - .bss i think)
it is uninitialized   
no need to make the EXE larger with a bunch of 0's

[sinsi]

In the section header there are two fields, VirtualSize (memory needed for the section) and SizeOfRawData (in the exe image).
When a section contains only uninitialized data, SizeOfRawData (and PointerToRawData) will be zero i.e. not in the image.

[Gunther]

Hi sinsi,

In the section header there are two fields, VirtualSize (memory needed for the section) and SizeOfRawData (in the exe image).
When a section contains only uninitialized data, SizeOfRawData (and PointerToRawData) will be zero i.e. not in the image.

I think that's true for the EXE image. But if the EXE is loaded into the RAM, the necessary space is allocated.

Gunther

[sinsi]

Hi Gunther, that's what VirtualSize is. As an added bonus the memory will be zeroed, no need to initialize variables to 0.

[Gunther]

Hi sinsi,

As an added bonus the memory will be zeroed, no need to initialize variables to 0.

good to know. Is that feature documented by MS?

Gunther

[jj2007]

Is that feature documented by MS?

Search old & new forum for uninitiali documented

Hint: You can safely assume that tons of professional and not-so-professional software would crash into your face if the .bss was not initialised to zero.

[K_F]

Well this is nice to know...
Was playing around (testing)  with a 4MB buffer...

My original EXE was just over 1MB... with the buffer in the .data? section, but just 'under' (?) 4MB in the .data section.
(This could explain bloatware that does so little for it's payload ??)

This can save me farting around with the global heap for a lot of known buffer sizes
Currently I use the heap to the tune of 80MBs worth of buffers and all the management is giving me a headache
:)

[K_F]

Hint: You can safely assume that tons of professional and not-so-professional software would crash into your face if the .bss was not initialised to zero.

This has always been my gripe with non-assembler programs (pro, or non-pro) - they never initialised to zero.
As I always say to the family... 'Never assume or think - make sure!!'
:)

[Gunther]

Jochen,

Hint: You can safely assume that tons of professional and not-so-professional software would crash into your face if the .bss was not initialised to zero.

I know that it's safe for the ELF format. I assume that MS made the specification here, but I'm not sure.

Gunther

[qWord]

I assume that MS made the specification here, but I'm not sure.

see Chapter 3: Section Table --> section headers --> members VirtualSize and SizeOfRawData

[dedndave]

Jochen and I had this discussion some time ago
i don't recall where we found it, but it is documented