Confusion about code, memory and addressing

[CCurl]

Ok, so here is my goal ...

I want to build a Forth system whose dictionary can grow indefinitely. In this implementation, I want it to be able to dynamically generate code at run time that can be executed during that same run time session. Since I want it to be able to grow indefinitely, I would like to use something like GlobalAlloc to allocate the memory space, and have the code that gets generated in that memory space.

So essentially, during run time, I want to allocate a big chunk of memory, and then dynamically create new code that I can call and also store data, all in that memory that I allocated.

Is something like this allowed? If so, then how you go about addressing that memory?

If not, then it's not the end of the world, I'll come up with something else, but it might not run as fast.

Thanks

[dedndave]

i would probably use the heap functions

GetProcessHeap
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366569%28v=vs.85%29.aspx

HeapAlloc
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366597%28v=vs.85%29.aspx

HeapFree
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366701%28v=vs.85%29.aspx

to allow code execution, you will probably have to use VirtualProtect to set the
attribute to PAGE_EXECUTE(_something)

VirtualProtect
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366898%28v=vs.85%29.aspx

Memory Protection Constants
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366786%28v=vs.85%29.aspx

[CCurl]

Thanks!

[rrr314159]

That's an ambitious project ...

Off the top of my head I'd think of getting the interpreter working first then the compiler. The thing is, when you dynamically generate code, into memory marked for execution, then jump into it, debugging is difficult (put it mildly). BTW, if you ever distribute the program, making executable segments will irritate anti-virus software; there are ways to deal with that. So I'd put it off as long as feasible, i.e. get a solid interpreter working first. Of course that's up to you.

Now, you don't mention needing to dynamically allocate memory? With a "data?" statement you can easily allocate a billion bytes (assuming you have a semi-modern PC) - wouldn't that be plenty for now? But GlobalAlloc (or, HeapAlloc is the more up-to-date way to do it) is no big deal, (except for making memory executable, which as I say can be dangerous). dedndave gave references for the msdn pages. masm32.lib (which is automatically available if you include \masm32include\masm32rt.inc) provides alloc and free functions which can be used as is, or studied to see how it works. You can check out the "alloc" function in \masm32\m32lib\alloc.asm. This is old-fashioned but works perfectly well. There are also examples of how to use the functions in masm32 directory.

GlobalAlloc, HeapAlloc, or masm32 alloc simply return a pointer to the memory in eax. Typically you'd store this somewhere, and typically use esi or edi to reference it, with statements like "mov [edi], myWord". If you want separate areas you'd define, e.g., "StartofDataStorage dd ?" and then "add edi, 1000000", then "mov StartofDataStorage, edi". And so on. You can also do this with data? statements.

If you know all this already, please ignore. If it doesn't address your concerns, sorry

Good Luck,

[jj2007]

Here is a plain Masm32 example for playing with the heap functions.

Code Select Expand

include \masm32\include\masm32rt.inc

.data?
pWorkArea dd ? ; pointer to work area
sizeWA dd ?
.code

start:
  print chr$(9, "1234567890123456789012345678901234567890123456789012345678901234567890", 13, 10)
  push edi
  mov sizeWA, 40

  mov pWorkArea, rv(HeapAlloc, rv(GetProcessHeap), HEAP_ZERO_MEMORY, sizeWA)
  mov ecx, sizeWA
  dec ecx ; you need one byte for the string's zero delimiter
  mov al, "x"
  mov edi, pWorkArea
  rep stosb ; simple memfill 40
  mov al, 0
  stosb ; put zero delimiter
  print "step 1", 9
  print pWorkArea, 13, 10

  sub sizeWA, 20 ; shrink your memory by 20 bytes
  mov pWorkArea, rv(HeapReAlloc, rv(GetProcessHeap), HEAP_ZERO_MEMORY, pWorkArea, sizeWA)
  mov edx, sizeWA
  mov edi, pWorkArea
  mov byte ptr [edi+edx], "_"
  print "step 2", 9
  print pWorkArea, 13, 10

  add sizeWA, 40 ; enlarge to 60
  mov pWorkArea, rv(HeapReAlloc, rv(GetProcessHeap), HEAP_ZERO_MEMORY, pWorkArea, sizeWA)
  print pWorkArea, 13, 110

  mov ecx, sizeWA
  dec ecx
  mov al, "."
  mov edi, pWorkArea
  add edi, 20 ; start at byte 20
  sub ecx, 20+1 ; correct loop count for offset, leave one byte for zero delimiter
  rep stosb ; simple memfill
  mov al, 0
  stosb
  print "step 3", 9
  inkey pWorkArea
  invoke HeapFree, rv(GetProcessHeap), 0, pWorkArea
  pop edi
  exit

end start

[dedndave]

there are a couple of instructions that are not easily relocated
they are specific forms of CALL and JMP

CALL (near, relative) (not all CALL's are relative)
JMP (near, relative) (not all JMP's are relative)
all conditional jumps (JE, JC, JA, etc) are near or short, and relative

relative operands are signed values, calculated as follows....

OperandValue = TargetAddress - AddressOfInstructionFollowingCALL

for short branches, that's a byte value
for near, it's a dword value

now, if the target address is also relocated, and the relative distance doesn't change,
you don't have to mess with it
but if either the target or the CALL are moved, you may have to deal with it   :P

here's a potential scenario that's not easily overcome.....

the code is located, and contains a conditional branch that is short (byte)
when relocated, the branch now needs a long operand
there isn't room for it in the code stream - oops !!!!

now, if you generate the code on the fly, you can avoid that problem
and, you will probably set off all kinds of bells and whistles with your AV program   :lol:

[dedndave]

by the way....

back in the late 80's and early 90's, i used to play a DOS game called Starflight
i was amazed at how much information the program seemed to hold and the total size was about 768 K or so
it has about 250 star systems and each system has 0 to 8 planets
you can land and move around on most of them - it seems like too much data would be required

so, out of curiosity, i disassembled parts of the code
it was written in Forth

https://en.wikipedia.org/wiki/Starflight#Development

what i found was, they had several small routines
each routine was processed by handling a series of "command strings"
the command strings were comprised of sequences of code addresses
the code processed the strings by using ESI and the LODS instruction
if a specific command wanted to process another command string,
it would PUSH ESI, load the address of a new command string, and execute, POP'ing ESI when done

so, to generate code, all you have to do is to generate a new sequence of command strings   
that should save you the trouble of having to generate machine language code from scratch

[CCurl]

The "list of addresses" solution is very common for Forth implementations. I am (was) thinking that it introduces overhead at run time, which would affect the performance, but perhaps the performance hit is small enough that I shouldn't worry about it.

I will start with that approach and see how it goes.

Thanks for all your thoughts and insights.