Strange Visual Studio problem

[KeepingRealBusy]

I have just encountered a very strange Visual Studio problem. My system is running Win 7. I am using  VS 2008 as a JIT debugger.

My program is assembled with MASM 8.0 (I tried both with MASM 8.0 and also Masm 9.0, both reacted the same way). This program is structured the same way as I structure my other programs. I have just checked another program and it works correctly. Only this one program fails.

The program structure (same structure as other programs, just different data):

Code Select Expand


INCLUDE <.\pkg\prog.pkg
               INCLUDE <.\inc\prog.inc
                        INCLUDE <.\inc\WINAPI
                        INCLUDE <.\inc\Runtime
                       etc.
               INCLUDE <.\pkg\DataDef.pkg
               .code
               main PROC
               int 3
               call GetMemory
               .
               .
               .


The int 3 is the first thing executed in the program proper. VS comes up, but instead of displaying the program package code, it is displaying the empty lines at the end of the DataDef.pkg. If I bring up the dissassembly window, VS properly displays the dissassembly code with the int 3 and the call GetMemory. As soon as I step into the GetMemory call, VS properly pops up the code window with the GetMemory source code. If I step out of the GetMemory code, I am back at the dissassembly window again. If I switch to the DataDef.pkg and step through the "code", VS walks through the empty lines at the end of the DataDef.pkg.

For some reason, MASM or Link or VS has forgotten where the main PROC code is (everything following the  INCLUDE <.\pkg\DataDef.pkg.

If I manually open the prog.pkg file in VS and try to set a breakpoint, I get the nasty message that there is no code associated with those locations so the breakpoint is disabled.

Has anyone else run into this? If so, what did you "fix" to get out of it?

Dave.

[GoneFishing]

Hi,
What is the command line for ML ?
Once I had much the same problem with WinDbg. It seems like the debugger knows about the associated source code files but can not see the lines' numbers.
I checked ML options and added /Zf switch to already existing /Zi.
HTH

Regards

[KeepingRealBusy]

vertograd

Thanks for the hint, but it did no good. Same problem. In addition, my other programs that work use the same ML options. I tried it anyway, no help.

Dave.

[Zen]

DAVE,
What in general terms is the content of the file: DataDef.pkg ???
I searched over at MSDN, and found this page: File Types Created for Visual C++ Projects. I have never seen a file with the .pkg extension in one of my Visual Studio project directories.

[KeepingRealBusy]

Zen,

.pkg is just a private extension. MASM doesn't care, it is just an INCLUDE of a file.extension. I use it to indicate a package of procedures or data definitions. I do not use libraries, just INCLUDES of my source (except for kernel32.lib), I roll my own runtime.

Everything (since 2001 when I started my MASM career) has run, until now with this program.

The data in the DataDef.pkg is nothing but data definitions (dxxxx DWORD 0) (szString BYTE "msg",10,13,0) etc.

Dave

[GoneFishing]

Can you post the content of Output (debug) window?

[KeepingRealBusy]

vertograd,

I have attached a zip of 4 screens. The first is the first interruption at the start of Main and it shows that VS thinks the code is at the end of the DataDef.pkg INCLUDE file. The second is with a disassembly window and it shows the actual code at the start of Main. The third is a disassembly window after I stepped into the Get Memory procedure which shows that VS knows about the GetMemory source file. The fourth is the code in GetMemory. If I step out of GetMemory and return to the caller, I end back at the disassembly window but there is no code window onlt the end of the DataDef.pkg.

Dave.

[GoneFishing]

Dave,
I can't see if the Output window is empty . Though it's likely to be.
If you say that :

This program is structured the same way as I structure my other programs. I have just checked another program and it works correctly. Only this one program fails.

The program structure (same structure as other programs, just different data):

I thought that the reason of the strange behaviour of the debugger might be in your code (its purpose , for example ). Don't you test some anti-debugging features, do you?

From GetMemory.pkg

;    Allocate all of memory except for a reserved block for JIT debugger.

[KeepingRealBusy]

vertograd,

I get the problem even before I allocate memory, right at Main entry.

My memory allocation always allocates the largest block possible (VirtualAlloc), and does this 10 times, then de-allocates a block thought to be large enough to bring in the debugger then does an int 3. If the program dies with no messages (instant death), then the reserved block is not large enough, double the size and try again until you get the breakpoint. If you get the breakpoint immediately, then reduce the reserved size by a half and try again until you fail (saving the last working size until failure. Now you have the lowest and highest size for success and failure, so try for the middle of that range and replace the lower limit if failure or the higher limit for success and try again until you get to a size such that that (high - low) is a system page size (4 KB). You now have the minimum size required by the debugger loading. Can that value and allocate all of memory and deallocate the canned reserve size from the smallest allocation (delete that allocation, allocate the reserved size, allocate the rest then delete the4 reserved size).

If there is any interest, I can supply a zip with my procedures, probably in the form of a working program that only starts by allocating all memory, then gets a huge number of large blocks, then takes an int 3 so you can examine the allocations. This is all 32 bit code, I have not tried this with 64 bit code.

I use this method for all of my memory needs.

Dave.

[GoneFishing]

Dave,
It does sound interesting but I must confess I'm  a newbie learning  MASM basics .

[Zen]

This whole thread confuses the hell out of me,...but, it seems like an interesting problem.
I have several questions:   

• Your program is failing for some unknown reason. It is written in assembly language and compiled with a version of MASM. You are using VS 2008 as a JIT debugger, which apparently launches when your program crashes. Why are you using a NET Framework JIT debugger for native code ?
• Why not use OllyDbg ??? Assembly programmers seem to prefer it over other inferior debuggers.
• Where exactly is the program failing ? Do you have exception handlers written and added to the exception handler chain ? (Visual Studio does this automatically, whereas MASM does not.)
• Have you tried to compile other versions of your program, using variations (and commenting out lines of code that could possibly contribute to the error) to accurately identify the problem ?

...The reason I'm asking all these annoying questions is that, it is well known that loading an executable into a debugger often alters the sequence of execution of the processor instructions. This is a very subtle error,...and extremely difficult to identify.

Of course, I could be wrong,...I almost always am,... :lol:

[KeepingRealBusy]

Zen,

• 
  My program is not failing. I inserted an int 3 at the entry to Main so I could step through the code. What is failing is Visual Studio when it is loaded to process the int 3 Exception.
  
• 
  I am using Visual Studio debugger because I am familiar with it. I tried OllyDbg but liked VS better.
  
• 
  The failure occurs right at the entry to the program. No code has been executed except for the int 3.
  
• 
  I have assembled the code with both MASM 8.0 and MASM 9.0. Both fail the same way.
  
• 
  I have removed all code and data from my program except my memory allocator and the very small data required for it. The stripped down version also has the VS error - brings up the data include file when it is, in fact, executing the first instruction in the code segment - the int 3. The debug window properly shows the code lines intermixed with lines from the DataDef include file.
  
  This same structure with different data works correctly for another of my programs with an inserted int 3 at the start of the program thus invoking VS.
  

Finally, I am not loading an executable into a debugger, it is loaded normally by the system and the debugger only gets involved when the int 3 is executed. There are no sequences of processor instructions that could be altered.

I am not positive, but I am now thinking that the problem may be that the code segment following the INCLUDE of the DataDef may be so small that it is causing problems??? All of the other programs that I use and that run correctly have large code segments.

Dave.

[japheth]

I have assembled the code with both MASM 8.0 and MASM 9.0. Both fail the same way.

You could try Masm v6.15. It writes the old COFF line-number debug info, unlike Masm 8 or 9, which write the new CV8 format. Actually, the new format is better organized and easier to handle, so one would think that if the new format fails, the probability for the old forma to fail is 150%, but you never know.

In case you need to use Masm 8+: I just uploaded the most up-to-date PE version - this tool allows to display CV8 debug info, including line-number info. The output is probably way too technical for most people, but it should at least give a hint if something is "corrupted". Of course, IF something is corrupted the question will arise what to do then - AFAIK MS is not going to fix errors in Masm ASAP.

[Zen]

DAVE,
Ahh Hah !!! It all makes sense to me now. I wondered what the int 3 instruction was there for.
So, you're looking at an exception, and Visual Studio is your registered debugger.   

...I googled the phrase: int 3 exception,...and, found this discussion at StackOverflow: Catching Opcode 0xCC as an Exception
The person that answers the inquiry states:  "Please note that a code running with a debugger attached will not see the exception, because the debugger handles it and clears it before passing the hand back to the code."
Here is the MSDN webpage about: Structured Exception Handling
...I seem to remember that EDGAR was involved in writing an assembly program to output the information from an exception handler, about a year ago,...and the question was so complicated that no one understood what he was talking about,...except EDGAR, of course.
Here is the link to that Old UK Forum thread: AddVectoredExceptionHandler Single Stepping, Mar 2011
If you search the Old UK MASM Forum (using the advanced search feature, check all) for the search term: exception,...and by user: donkey,...you will find a number of Forum posts (EDGAR often writes in the GoAsm forum) that deal with exception handling in MASM.
Here is an example of typical EDGAR genius: Monitoring a Memory Address, May 2011,...
or: Breakpoint problem, Aug 2009
...or, the mind-boggling EDGARism: Reading Raw Debug Symbol Data, Mar 2010

...Thank GOD all the Canadians aren't as smart as EDGAR,...or, they would have taken over the world long ago,...

...What do you think ?

[jj2007]

code running with a debugger attached will not see the exception, because the debugger handles it and clears it before passing the hand back to the code.

What happens if Olly is your JIT is that Windows greets you with a fat MsgBox; if you click Cancel or hit Escape, Olly takes over and says "INT 3 command at ... ". You can press F8 to continue stepping through the code.

Olly shields you from going through KiUserExceptionDispatcher, that's all.

P.S.: EDGAR has recently been a bit silent, therefore here an example of SEH in Genuine MASM^TM:

include \masm32\MasmBasic\MasmBasic.inc        ; download
  Init tc                        ; initialise MasmBasic with Try/Catch (aka SEH)
  xor ecx, ecx
  Try
        div ecx        ; do something illegal
        MsgBox 0, Str$("div succeeded, eax is %i\n", eax), "SEH Test:", MB_OK  ; you won't see that box
  Catch Only
        MsgBox 0, LastEx(info), "SEH in Masm32: div ecx failed", MB_OK
  Finally
  Exit
  TryCatchEnd
end start

Output:
{ERRORE DI EXCEPTION}
Divisione intera per zero.
EIP     0040104B
Code    C0000094

And don't blame me for translating "integer division" with "entire division", it's our friends in Redmond who can't afford professional translators.