Memory-mangement dilemma

[zedd151]

I've never seen Notepad not be able to load a huge file, so it must have some feedback going on as it works; maybe it can detect that it's hit the end of its buffer and just silently reallocates it. I don't see how to do this with a regular edit control; I guess you could look for EN_CHANGE notifications, which happen whenever the user types a character, and constantly check the amount of text in the control against the text limit and adjust it as needed ahead of time.

I've haven't looked under the hood of notepad for a long time. All this time I thought notepad used a rich edit control, when I read this I went and looked if it loads richedit. Sure enough a plain vanilla EDIT control.

Please translate to English

Don't you just love Magnus' late night rambling posts, after he's been hitting the sauce?    Not that I know that that is true though. Sorry Magnus if it is not, but the rambling makes it appear so.

@ NoCforMe - Consider using two buffers? A very large file sized buffer and a smaller edit window sized buffer? When scrolling could read from larger file  buffer to smaller edit window buffer (depending on how many lines were scrolled and how many lines of text are visible)? Involves tokenizing the lines of text that much I do remember. Also involves incrementing index to the tokenized lines of text.
I've used a similar technique in the past, looking for that code now. iirc the code for doing that is a little messy and somewhat cumbersome but iirc makes scrolling a huge file faster.

edited to add:
Later:
I'm going to do some tests using a method as just described later today when I've got some spare time...


Later yet::
Okay, in the examples folder of the Masm32 SDK is an example "qikpad" which uses an EDIT control.
It can open & save 100 MB file without issue  (after removing the 32kb limit warning) - modifed version of qikpad is attached. Used 100 x "winextra.inc" = 100 MB for testing purposes. So much for the alleged 32kb limit. They must have changed the internals of EDIT controls since the Windows 98 days??


So, in light of that will not be trying the method I suggested earlier.

[zedd151]

from MSDN

Remarks
The EM_SETLIMITTEXT message limits only the text the user can enter.
It does not affect any text already in the edit control when the message is sent, nor does it affect the length of the text copied to the edit control by the WM_SETTEXT message.
If an application uses the WM_SETTEXT message to place more text into an edit control than is specified in the EM_SETLIMITTEXT message, the user can edit the entire contents of the edit control.
Before EM_SETLIMITTEXT is called, the default limit for the amount of text a user can enter in an edit control is 32,767 characters.

Ah, ok. The limit is for ENTERING TEXT, not setting window text via WM_SETTEXT or SetWindowText. 


I just tried adding text to that 100 MB file. no luck    will not allow more chars to be added there.    So, nevermind. 

[jj2007]

Code Select Expand

GuiParas equ "Hello World", x300, w700, h999, b LiteBlueGreen ; width+height, margins, background colour
include \masm32\MasmBasic\Res\MbGui.asm
  NanoTimer()
  ClearLastError
  GuiControl MyEdit, "Edit", bcol LiteYellow, fcol Black, font -14 ; colours & fonts have no effect with rtf files
  invoke SendMessage, hMyEdit, EM_SETLIMITTEXT, 5000000, 0
  SetWin$ hMyEdit=FileRead$(CL$())
  GuiControl MyStatus, "statusbar", wRec$(NanoTimer$())
GuiEnd

So far, so simple: drag a file over the exe and see it load. What's interesting is that a 4.4MB "bible.txt" takes about 19 seconds to load. Notepad is a bit faster, at 4-5 seconds - how can it be 4 times as fast? Secret switches?
What's even more interesting is that, if you change "Edit" to "RichEdit", the same file loads in ... taraa! ... 105ms. That's a factor 176 faster. I like it.

Executables are attached.

[zedd151]

from MSDN

Remarks
The EM_SETLIMITTEXT message limits only the text the user can enter.
It does not affect any text already in the edit control when the message is sent, nor does it affect the length of the text copied to the edit control by the WM_SETTEXT message.
If an application uses the WM_SETTEXT message to place more text into an edit control than is specified in the EM_SETLIMITTEXT message, the user can edit the entire contents of the edit control.
Before EM_SETLIMITTEXT is called, the default limit for the amount of text a user can enter in an edit control is 32,767 characters.

Ah, ok. The limit is for ENTERING TEXT, not setting window text via WM_SETTEXT or SetWindowText. 


I just tried adding text to that 100 MB file. no luck    will not allow more chars to be added there.    So, nevermind. 

But this works:

Code Select Expand

EditML proc szMsg:DWORD, tx:DWORD, ty:DWORD, wd:DWORD, ht:DWORD,
            hParent:DWORD, ID:DWORD, Wrap:DWORD
    LOCAL hCtl   :DWORD
    LOCAL hFnt   :DWORD
    LOCAL eStyle :DWORD
    szText CtlStyle, "EDIT"
    mov eStyle, WS_VISIBLE or WS_CHILDWINDOW or \
                WS_VSCROLL or ES_NOHIDESEL or \
                ES_AUTOVSCROLL or ES_MULTILINE
    .if Wrap == 0
      or eStyle, WS_HSCROLL or ES_AUTOHSCROLL
    .endif
    invoke CreateWindowEx, WS_EX_CLIENTEDGE, ADDR CtlStyle, szMsg,
                          eStyle, tx, ty, wd, ht, hParent, ID, hInstance, NULL
    mov hCtl, eax
    invoke SendMessage, hCtl, EM_SETLIMITTEXT, 0, 1024000000   ;  <---- 1GB you won't need more I hope :P
    invoke GetStockObject, SYSTEM_FIXED_FONT
    mov hFnt, eax
    invoke SendMessage, hCtl, WM_SETFONT, hFnt, TRUE
    mov eax, hCtl
    ret
EditML endp

Now can add text to very large file. Edited source attached
Apparently internally some form of memory management is already done - to allow more text to be written to the control. Only time that you need worry about memory size is when copying the text to a buffer for writing to file, which is a simple task.
Why on earth in these days of ample memory are the EDIT controls limited at all? I understand in the early days of x86 memory was limited by hardware constraints, and other factors. In any case I mostly use richedit controls with all of their idiosyncrasies, as they are more versatile - BUT still use EDIT in dialog boxes for their simplicity of usage there. edit = clarification.

[NoCforMe]

Api Get file size + extra buffer= max typed chars in a day by fast secretary before load file choose by user?
Biggest file size change copy/paste big source file, api get clipboard size and reallocate + clip size?

Please translate to English

I get the gist of what he's suggesting; somehow estimate the amount of text likely to be added to a file after being loaded, including extra text pasted to it. I have no idea how to get such an estimate. The technical staff here at NoCforMe Laboratories, GmbH, isn't able to derive such an estimate.

I'll just ballpark it, I guess.

[hutch--]

Z,

qikpad was about the last time I wrote a text editor using the standard edit control. With a rich edit 2-3 control in 32 bit, 100 meg is no problem, in 64 bit over 700 meg can be done. If you set the limit to "-1" you can load very large text.

RTF is a different animal, it is much more complex and does much more work so the speed drops off a lot faster. I generally use a normal edit control for dialogs or where relatively small amounts of text are used and use the rich edit controls for large text and drag and drop, both of which works well.

[zedd151]

I'll just ballpark it, I guess.

use WM_GETTEXTLENGTH or a call to GetWindowTextLength??? to determine buffer size needed...

[zedd151]

Z,...

qikpad was about the last time I wrote a text editor using the standard edit control. 

Understood of course, written back in the stone age.  I was just using it as an example since NoCforMe is also using an EDIT control for this project.

[jj2007]

use WM_GETTEXTLENGTH??? to determine buffer size needed...

When you load a file, you know its size.

[zedd151]

When you load a file, you know its size.

For the modified, i.e., 'added to' size 
After adding text, the loaded filesize is meaningless unless you are just opening a file to turn right around and close it.

@ NoCforMe...
If you want to pre-allocate (as suggested in original post) you will of course risk a buffer overrun, which I assume is undesirable.

[NoCforMe]

When you load a file, you know its size.

For the modified, i.e., 'added to' size 
After adding text, the loaded filesize is meaningless unless you are just opening a file to turn right around and close it.

Right, so you haven't answered the question at all, which is "how much extra should I add to the buffer size after opening the file?". Of course I know the size of the file and therefore the minimum buffer size needed to hold it, but that value needs to be increased or the poor user won't be able to add any new text!

@ NoCforMe...
If you want to pre-allocate (as suggested in original post) you will of course risk a buffer overrun, which I assume is undesirable.

No, that's not an issue here; the edit control code takes care of that. The buffer won't be overrun; the control simply won't let you add anything extra to it, which is what happened to me when the text limit was less than the size of the file I loaded. It didn't overrun the buffer, just refused any input.

[daydreamer]

Api Get file size + extra buffer= max typed chars in a day by fast secretary before load file choose by user?
Biggest file size change copy/paste big source file, api get clipboard size and reallocate + clip size?

Please translate to English

I get the gist of what he's suggesting; somehow estimate the amount of text likely to be added to a file after being loaded, including extra text pasted to it. I have no idea how to get such an estimate. The technical staff here at NoCforMe Laboratories, GmbH, isn't able to derive such an estimate.

I'll just ballpark it, I guess.

Something like that
Many years ago I tried find solution to use rich edit control as alternative to achieve similar as console loc column, row macro combined with print for simple output tictactoe x's and o's or other simple ascii or unicode based output, I couldn't figure out characters / line,so i could make a loc column,row function or macro for rich edit 
Just change text buffer and SendMessage to rich edit
Anyone made a function to go certain line number before?

[hutch--]

 

Don't make DOS assumptions, add a megabyte and see if that extends your limit, then try 10 megabytes etc ....

[zedd151]

but that value needs to be increased or the poor user won't be able to add any new text!

Not really, internally it is handled. You only need to then retrieve text length AFTER text has been added. You don't need to use the same buffer after writing more text into the edit window. You already know this stuff, so whats the dilemma?
Thats all I have to say here. 

[jj2007]

Right, so you haven't answered the question at all, which is "how much extra should I add to the buffer size after opening the file?"

That answer has been given at least twice: invoke SendMessage, hEdit, EM_SETLIMITTEXT, -1, 0