Author Topic: Maximum size for Readfile/Writefile (ex) Functions  (Read 3607 times)

K_F

  • Member
  • *****
  • Posts: 1287
  • Anybody out there?
Maximum size for Readfile/Writefile (ex) Functions
« on: January 04, 2017, 03:10:43 AM »
As it is in the title....

I'd like to read write a file anywhere up to 1GB, all in one go = A big fat data file
Can these functions handle it ?
 :biggrin:
'Sire, Sire!... the peasants are Revolting !!!'
'Yes, they are.. aren't they....'

Adamanteus

  • Member
  • **
  • Posts: 180
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #1 on: January 04, 2017, 04:51:58 AM »
 System functions reading/writing not more, than 16K of data - for more is need sequental calls of them.

jj2007

  • Member
  • *****
  • Posts: 7548
  • Assembler is fun ;-)
    • MasmBasic
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #2 on: January 04, 2017, 05:30:32 AM »
System functions reading/writing not more, than 16K of data - for more is need sequental calls of them.

That's an interesting theory - where did you read that?

include \masm32\MasmBasic\MasmBasic.inc      ; download
  Init
  Let esi=FileRead$("\Masm32\MasmBasic\AscUser\Bible400.txt")
  Inkey Str$("%i bytes read in one go", LastFileSize)
EndOfCode


Output:
Code: [Select]
1618956800 bytes read in one go

Adamanteus

  • Member
  • **
  • Posts: 180
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #3 on: January 04, 2017, 07:29:40 AM »
And FileRead$ - system function, or it have quantum blocks algo ?

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 4809
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #4 on: January 04, 2017, 08:05:12 AM »
Van,

The only real limit is the amount of memory you can allocate for very large files in 32 bit mode. While the guts of ReadFile/WriteFile appear to stream data sequentially, they handle pointing the stream to a single block of memory.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :biggrin:

jj2007

  • Member
  • *****
  • Posts: 7548
  • Assembler is fun ;-)
    • MasmBasic
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #5 on: January 04, 2017, 08:23:41 AM »
And FileRead$ - system function, or it have quantum blocks algo ?

What do you think "read in one go" means? Btw you have not answered my question "where did you read that?" 8)

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #6 on: January 04, 2017, 09:43:18 AM »
And FileRead$ - system function, or it have quantum blocks algo ?

What do you think "read in one go" means? Btw you have not answered my question "where did you read that?" 8)

The maximum size of a standard IO stream is one page.
Code: [Select]
#define _MAXIOBUF 0x4000
#define _INTIOBUF 0x1000
#define _MINIOBUF 0x0200

The maximum size of ReadFile() depends on the API version.
Quote
ReadFile
This function entry is part of a MAPI-specific list of Win32 functions. For a complete description, see Win32 Functions.
BOOL ReadFile(
  HANDLE hFile,             
  LPVOID lpBuffer,           
  DWORD NumberOfBytesToRead,   
  LPDWORD lpNumberOfBytesRead,   
  LPOVERLAPPED lpOverlapped 
)
Limitations
The count is limited to 64K. The lpOverlapped parameter is not supported and the function fails if lpOverlapped is not NULL.
 
However, using a buffer above 16K is a bit over the top for many reasons so that should be avoided if possible.


jj2007

  • Member
  • *****
  • Posts: 7548
  • Assembler is fun ;-)
    • MasmBasic
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #8 on: January 04, 2017, 12:58:25 PM »
The maximum size of ReadFile() depends on the API version

We are not talking Windows 3.1 here; even on my Win XP VM, I can read 10 MB into a buffer

Quote
using a buffer above 16K is a bit over the top for many reasons so that should be avoided if possible.

I can't see a single good reason why one shouldn't fill a buffer in one go. Of course, 1.6GB as in the Win7 example is pretty fat, and you should not rely on it to work on all systems, but 16k as the maximum? Ridiculous.

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 4809
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #9 on: January 04, 2017, 02:33:33 PM »
Van,

Just one extra suggestion if this is in 32 bit, set the /LARGEADDRESSAWARE option on your linker command line so that you can safely allocate large amounts of memory.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :biggrin:

Adamanteus

  • Member
  • **
  • Posts: 180
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #10 on: January 04, 2017, 04:57:25 PM »
What do you think "read in one go" means? Btw you have not answered my question "where did you read that?" 8)
Limitations
The count is limited ...
- I read manual till this, after it read was simle lazy - six theare was, one I myself added, so 16 ...

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #11 on: January 04, 2017, 07:45:14 PM »
In all fairness to the question the correct answer is provided by hutch: the only real limit is the amount of memory you can allocate.

Quote
I can't see a single good reason why one shouldn't fill a buffer in one go.

Agree on that one.

Quote
Of course, 1.6GB as in the Win7 example is pretty fat, and you should not rely on it to work on all systems

True.

Quote
but 16k as the maximum? Ridiculous.

Also true. The maximum should be 4K.

jj2007

  • Member
  • *****
  • Posts: 7548
  • Assembler is fun ;-)
    • MasmBasic
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #12 on: January 04, 2017, 08:13:29 PM »
In all fairness to the question the correct answer is provided by hutch: the only real limit is the amount of memory you can allocate.

This is correct.

Quote
The maximum should be 4K.

This is nonsense.

If you need a 100MB buffer e.g. to analyse a big database, you allocate the 100MB and ask ReadFile() to fill it. In one single operation. Splitting that operation in a loop where you read 25,000 times your 4k buffer, and copy the content 25,000 times to the main buffer is probably what they exercise in certain university courses to explain what a "loop" is, but it's definitely not efficient code :eusa_snooty:

Please do not forget that we are in the Campus here - n00bs may read this stuff and actually believe it. This is why I corrected Adamanteus' reply #1, and asked where he had read that. He just answered "I read manual" but no link so far.

Your reply ("This function entry is part of a MAPI-specific list of Win32 functions...") has quotes but no URL to the source. According to Google, this language was last used 17 years ago...

So, coming back to the OP's question:

I'd like to read write a file anywhere up to 1GB, all in one go = A big fat data file
Can these functions handle it ?

YES.

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 4809
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #13 on: January 04, 2017, 09:19:48 PM »
The test is really simple, choose a 1 gig VOB file from a DVD, get its exact length, call your preferred memory allocation strategy, check the return value, if its an error (IE: not enough memory) tell the user, if not, read the file into the memory in one pass. There are options to get under the normal file input that rely on direct hardware capacities but they are a pain to use, don't seem to be any faster and may not be universally supported on all OS versions.

My only recommendation is ensure you set the linker option /LARGEADDRESSAWARE so you get the most Win32 memory, matters less on Win7 64 but on Win 10 64 with more chyte loaded into Win32 memory by the OS, you may not have enough memory by default to handle large allocations.

There is a trick on a Win 64 system if it has enough memory, create a 1 gig memory mapped file that is shared between 32 bit and 64 bit applications. On the 64 bit end, use system messages to run the data from the memory mapped file to normal memory storage and have multiple memory slots available and at 1 gig at a time, you can store far more information that Win32 can handle. It works of a combination of messages sent with the HWND_BROADCAST handle and using the memory mapped file as the transfer vehicle. Its messy but it works, its almost the same logic as the DOS era expanded memory (switch banks) but with memory copy in excess of 2 gig/sec on modern hardware, it is a viable technique.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :biggrin:

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Maximum size for Readfile/Writefile (ex) Functions
« Reply #14 on: January 04, 2017, 09:41:52 PM »
Quote
This is nonsense.

 :biggrin:

Quote
If you need a 100MB buffer e.g. to analyse a big database,  you allocate the 100MB and ask ReadFile() to fill it. In one single operation.

Strange then that no major software companies seems to end up with this particular need.

Quote
Splitting that operation in a loop where you read 25,000 times your 4k buffer, and copy the content 25,000 times to the main buffer is probably what they exercise in certain university courses to explain what a "loop" is, but it's definitely not efficient code :eusa_snooty:

Think you find that all software produced by Microsoft and all other major software companies actually use a 4K IO buffer as standard in all applications but still manage to read and write files larger than 100M anyway. And even worse, the 4K IO-buffer is actually the main buffer.

Strange all of this, but if you have attended one of these university courses they may have explained how all of this magic works. There clearly must be a conspiracy going on here given the hardware companies seems to be into it too.

Quote
Please do not forget that we are in the Campus here - n00bs may read this stuff and actually believe it. This is why I corrected Adamanteus' reply #1, and asked where he had read that. He just answered "I read manual" but no link so far.

But what are we supposed to do if the universities, software companies, and hardware companies conspire against us with magic and produce all this propaganda?

Quote
Your reply ("This function entry is part of a MAPI-specific list of Win32 functions...") has quotes but no URL to the source. According to Google, this language was last used 17 years ago...

Lets see..
Code: [Select]
C:\masm32\bin>ml
Microsoft (R) Macro Assembler Version 6.14.8444
Copyright (C) Microsoft Corp 1981-1997.  All rights reserved.