Ok, so here is my goal ...
I want to build a Forth system whose dictionary can grow indefinitely. In this implementation, I want it to be able to dynamically generate code at run time that can be executed during that same run time session. Since I want it to be able to grow indefinitely, I would like to use something like GlobalAlloc to allocate the memory space, and have the code that gets generated in that memory space.
So essentially, during run time, I want to allocate a big chunk of memory, and then dynamically create new code that I can call and also store data, all in that memory that I allocated.
Is something like this allowed? If so, then how you go about addressing that memory?
If not, then it's not the end of the world, I'll come up with something else, but it might not run as fast.
Thanks
i would probably use the heap functions
GetProcessHeap
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366569%28v=vs.85%29.aspx (https://msdn.microsoft.com/en-us/library/windows/desktop/aa366569%28v=vs.85%29.aspx)
HeapAlloc
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366597%28v=vs.85%29.aspx (https://msdn.microsoft.com/en-us/library/windows/desktop/aa366597%28v=vs.85%29.aspx)
HeapFree
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366701%28v=vs.85%29.aspx (https://msdn.microsoft.com/en-us/library/windows/desktop/aa366701%28v=vs.85%29.aspx)
to allow code execution, you will probably have to use VirtualProtect to set the
attribute to PAGE_EXECUTE(_something)
VirtualProtect
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366898%28v=vs.85%29.aspx (https://msdn.microsoft.com/en-us/library/windows/desktop/aa366898%28v=vs.85%29.aspx)
Memory Protection Constants
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366786%28v=vs.85%29.aspx (https://msdn.microsoft.com/en-us/library/windows/desktop/aa366786%28v=vs.85%29.aspx)
Thanks!
That's an ambitious project ...
Off the top of my head I'd think of getting the interpreter working first then the compiler. The thing is, when you dynamically generate code, into memory marked for execution, then jump into it, debugging is difficult (put it mildly). BTW, if you ever distribute the program, making executable segments will irritate anti-virus software; there are ways to deal with that. So I'd put it off as long as feasible, i.e. get a solid interpreter working first. Of course that's up to you.
Now, you don't mention needing to dynamically allocate memory? With a "data?" statement you can easily allocate a billion bytes (assuming you have a semi-modern PC) - wouldn't that be plenty for now? But GlobalAlloc (or, HeapAlloc is the more up-to-date way to do it) is no big deal, (except for making memory executable, which as I say can be dangerous). dedndave gave references for the msdn pages. masm32.lib (which is automatically available if you include \masm32include\masm32rt.inc) provides alloc and free functions which can be used as is, or studied to see how it works. You can check out the "alloc" function in \masm32\m32lib\alloc.asm. This is old-fashioned but works perfectly well. There are also examples of how to use the functions in masm32 directory.
GlobalAlloc, HeapAlloc, or masm32 alloc simply return a pointer to the memory in eax. Typically you'd store this somewhere, and typically use esi or edi to reference it, with statements like "mov [edi], myWord". If you want separate areas you'd define, e.g., "StartofDataStorage dd ?" and then "add edi, 1000000", then "mov StartofDataStorage, edi". And so on. You can also do this with data? statements.
If you know all this already, please ignore. If it doesn't address your concerns, sorry
Good Luck,
Here is a plain Masm32 example for playing with the heap functions.
include \masm32\include\masm32rt.inc
.data?
pWorkArea dd ? ; pointer to work area
sizeWA dd ?
.code
start:
print chr$(9, "1234567890123456789012345678901234567890123456789012345678901234567890", 13, 10)
push edi
mov sizeWA, 40
mov pWorkArea, rv(HeapAlloc, rv(GetProcessHeap), HEAP_ZERO_MEMORY, sizeWA)
mov ecx, sizeWA
dec ecx ; you need one byte for the string's zero delimiter
mov al, "x"
mov edi, pWorkArea
rep stosb ; simple memfill 40
mov al, 0
stosb ; put zero delimiter
print "step 1", 9
print pWorkArea, 13, 10
sub sizeWA, 20 ; shrink your memory by 20 bytes
mov pWorkArea, rv(HeapReAlloc, rv(GetProcessHeap), HEAP_ZERO_MEMORY, pWorkArea, sizeWA)
mov edx, sizeWA
mov edi, pWorkArea
mov byte ptr [edi+edx], "_"
print "step 2", 9
print pWorkArea, 13, 10
add sizeWA, 40 ; enlarge to 60
mov pWorkArea, rv(HeapReAlloc, rv(GetProcessHeap), HEAP_ZERO_MEMORY, pWorkArea, sizeWA)
print pWorkArea, 13, 110
mov ecx, sizeWA
dec ecx
mov al, "."
mov edi, pWorkArea
add edi, 20 ; start at byte 20
sub ecx, 20+1 ; correct loop count for offset, leave one byte for zero delimiter
rep stosb ; simple memfill
mov al, 0
stosb
print "step 3", 9
inkey pWorkArea
invoke HeapFree, rv(GetProcessHeap), 0, pWorkArea
pop edi
exit
end start
there are a couple of instructions that are not easily relocated
they are specific forms of CALL and JMP
CALL (near, relative) (not all CALL's are relative)
JMP (near, relative) (not all JMP's are relative)
all conditional jumps (JE, JC, JA, etc) are near or short, and relative
relative operands are signed values, calculated as follows....
OperandValue = TargetAddress - AddressOfInstructionFollowingCALL
for short branches, that's a byte value
for near, it's a dword value
now, if the target address is also relocated, and the relative distance doesn't change,
you don't have to mess with it
but if either the target or the CALL are moved, you may have to deal with it :P
here's a potential scenario that's not easily overcome.....
the code is located, and contains a conditional branch that is short (byte)
when relocated, the branch now needs a long operand
there isn't room for it in the code stream - oops !!!!
now, if you generate the code on the fly, you can avoid that problem
and, you will probably set off all kinds of bells and whistles with your AV program :lol:
by the way....
back in the late 80's and early 90's, i used to play a DOS game called Starflight
i was amazed at how much information the program seemed to hold and the total size was about 768 K or so
it has about 250 star systems and each system has 0 to 8 planets
you can land and move around on most of them - it seems like too much data would be required
so, out of curiosity, i disassembled parts of the code
it was written in Forth
https://en.wikipedia.org/wiki/Starflight#Development (https://en.wikipedia.org/wiki/Starflight#Development)
what i found was, they had several small routines
each routine was processed by handling a series of "command strings"
the command strings were comprised of sequences of code addresses
the code processed the strings by using ESI and the LODS instruction
if a specific command wanted to process another command string,
it would PUSH ESI, load the address of a new command string, and execute, POP'ing ESI when done
so, to generate code, all you have to do is to generate a new sequence of command strings :biggrin:
that should save you the trouble of having to generate machine language code from scratch
The "list of addresses" solution is very common for Forth implementations. I am (was) thinking that it introduces overhead at run time, which would affect the performance, but perhaps the performance hit is small enough that I shouldn't worry about it.
I will start with that approach and see how it goes.
Thanks for all your thoughts and insights.