Bug in NTVDM.INC/LIB in MASM 11 and 10

[pcMike]

Hi Hutch,

Back in 2007 I let you know about an issue with the NTVDM.INC and MTVDM.LIB in MASM 10 here:
http://www.masmforum.com/board/index.php?topic=10004.0

[Edit]At the time I thought the problem was due to an incorrect MGetVdmPointer PROTO and Library.

Mine:   MGetVdmPointer PROTO :DWORD,:DWORD,:CHAR
Yours:     MGetVdmPointer PROTO :DWORD,:DWORD,:DWORD

Today I finally got around to upgrading to MASM32 11, and I see this problem was never fixed.

I confirmed that both the NTVDM.INC and NTVDM.LIB included with v11 have the same issue. I am attaching the working NTVDM.LIB which I have been using for the past 15 years. When I replaced your v11 .LIB file with mine, and edited your v11 NTVDM.INC to correct the MgetVdmPointer PROTO, my program works fine again.

Also, I noticed that when I attempted to install version 11 on Windows 10 32-bit, I got several error messages during the install, such as:
The ordinal 201 could not be located in the dynamic link library: C:\Windows\AppPatch\AcGenral.DLL
The ordinal 201 could not be located in the dynamic link library: C:\Windows\AppPatch\AcLayers.DLL
There were a few more of these, but I did not write them down.

Regards,  Mike

[TouEnMasm]

Hello,
a char is a BYTE.
in 32 bits PROTO :BYTE must be interpreted as "push Byte".
Push byte don't exist and masm re-interpret this to push a dword.
The error is elsewhere,made a listing of your source code and you will see what happen.
in your lib
     3FE2 _MGetVdmPointer@12                ;12 =3 * sizeof dword
     3FE2 __imp__MGetVdmPointer@12

[zedd151]

hutch:

...MASM does not have bugs....

(talking about another subject)

from here:
http://masm32.com/board/index.php?topic=3958.msg41686#msg41686

:P

[jj2007]

Hi Mike,

Welcome back :icon14:

This is so exotic that it probably just got forgotten. I would have needed that function ages ago when I still worked with 16-bit GfaBasic...

Googling reveals few hits for MGetVdmPointer. One is apparently a very old Microsoft source (here), with this definition:

PBYTE
MGetVdmPointer(
ULONG    Address,
ULONG    Size,
UCHAR   ProtectedMode
);

As ToutEnMasm wrote above, this may be interpreted as push byte - a no-no in Win32. Infact, the assemblers I have tested do NOT push a byte - they push a dword. This is not a problem when you use, in the snippet below,

Code Select Expand

invoke pb, 1, 2, 3

The 3 gets pushed as the DWORD 00000003h

But look at the second invoke:
  mov eax, 12345678h
  invoke pb, 1, 2, al

According to the proc definition, 00000078h should be pushed, e.g. using

Code Select Expand

movzx eax, al
push eax

But all assemblers tested (ML 6.14, 6.15 and 10.0, JWasm) simply push eax - so you have 12345678h on the stack... now find out what exactly implies for your code ;-)

Code Select Expand

include \masm32\include\masm32rt.inc
.code
pb proc arg1, arg2, arg3:UCHAR
  ret
pb endp

start:
  invoke pb, 1, 2, 3
  mov eax, 12345678h
  invoke pb, 1, 2, al
  exit
end start

[pcMike]

Hi ToutEnMasm and JJ,

You are correct, the problem was not the difference in NTVDM.INC. I changed the CHAR back to a DWORD, and the result was the same either way, so it has to be something different in NTVDM.LIB which is causing the issue. Now that I took a closer look, it don't think it's even related to the mGetVdmPointer routine at all, as the failure occurs before hand.

I don't remember how I acquired my original working NTVDM.LIB, but just now I tried using inc2l.exe to recreate the NTVDM.LIB (under Windows 10), and it fails the same in the same way as the one included with MASM32 v11, and it is also the same file size.

I'll explain a little about the issue itself.
I have a DOS application that uses the NTVDM "BOP" functions to interface with a Win32 Virtual Device Driver .DLL file. Details on how this is done can be found here:
http://www.ragestorm.net/tutorial?id=27

When I use the version of NTVDM.LIB that comes with MASM32 10 or 11 to build my .DLL file, the BOP function that registers the .DLL file fails with the result AX=1, which means "Could not find .DLL file", even though it is in the \windows\system32\ directory as it should be. But when I build the same .DLL using my older NTVDM.LIB, it works fine.

Here is a snippit of the DOS stub which loads the DLL:

Code Select Expand


;  The following BOP Macros were taken from ISVBOP.H
;  They allow a DOS app to interface with a Win32 DLL.
;  See ISVBOP.H and isvbop.inc for more information.

BOP_3RDPARTY    EQU     58h
BOP_UNSIMULATE  EQU     FEh
.RegisterModule macro   
db      0C4h, 0C4h, BOP_3RDPARTY, 0
endm
.UnRegisterModule macro
db      0C4h, 0C4h, BOP_3RDPARTY, 1
endm
.DispatchCall   macro   
db      0C4h, 0C4h, BOP_3RDPARTY, 2
endm

; ** RegisterModule - This Bop call is made from the 16 bit module
; *                   to register a third party DLL with the bop
; *                   manager. This call returns a handle to the
; *                   16bit caller which is to be used later to
; *                   dispatch a call to the DLL.
; *
; *  INPUT:
; *      Client DS:SI - asciiz string of DLL name.
; *      Client ES:DI - asciiz string of Init Routine in the DLL. (Optional)
; *      Client DS:BX - asciiz string to Dispatch routine in the DLL.
; *
; *  OUTPUT:
; *      SUCCESS:
; *          Client Carry Clear
; *          Client AX = Handle (non Zero)
; *      FAILURE:
; *          Client Carry Set
; *          Client AX = Error Code
; *                 AX = 1 - DLL not found
; *                 AX = 2 - Dispatch routine not found.
; *                 AX = 3 - Init Routine Not Found
; *                 AX = 4 - Insufficient Memory
; *
; *  NOTES:
; *      RegisterModule results in loading the DLL (specified in DS:bp).
; *      Its Init routine (specified in ES:DI) is called. Its Dispatch
; *      routine (specified in DS:BX) is stored away and all the calls
; *      made from DispatchCall are dispacthed to this routine.
; *      If ES and DI both are null than the caller did'nt supply the init
; *      routine.
;
mov     si,offset dllname_text
mov     bx,offset dispatchname_text
xor     ax,ax
xor     di,di
mov     es,di
.RegisterModule                 ; BOP Register Modual
jnc     VDD_loaded              ; CF=0 indicates success
cmp     ax,1
jne     VDD_chk1
mov    dx,offset msg_file_not_found     ; 'Can't find .dll'
jmp     short show_error


In case it helps, here is a exports dump of my original NTVDM.LIB

Code Select Expand


c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC>dumpbin /exports \users\mike\ntvdm.lib
Microsoft (R) COFF/PE Dumper Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file \users\mike\ntvdm.lib

File Type: LIBRARY

     Exports

       ordinal    name

                  _BlockWOWIdle@4
                  _CurrentMonitorTeb
                  _DBGNotifyDebugged@4
                  _DBGNotifyNewTask@8
                  _DBGNotifyRemoteThreadAddress@8
                  _DispatchInterrupts@0
                  _ExpLdt
                  _ExpVdmTib
                  _FlatAddress
                  _GetDOSAppName@4
                  _GetWOWShortCutInfo@8
                  _IsCdRomFile@4
                  _MGetVdmPointer@12
                  _RegisterWOWIdle@0
                  _ResumeTimerThread@0
                  _SelectorLimit
                  _Sim32pGetVDMPointer@8
                  _SoftPcEoi@8
                  _SuspendTimerThread@0
                  _VDDAllocMem@12
                  _VDDAllocateDosHandle@12
                  _VDDAssociateNtHandle@12
                  _VDDDeInstallIOHook@12
                  _VDDDeInstallMemoryHook@12
                  _VDDDeInstallUserHook@4
                  _VDDExcludeMem@12
                  _VDDFreeMem@12
                  _VDDIncludeMem@12
                  _VDDInstallIOHook@16
                  _VDDInstallMemoryHook@16
                  _VDDInstallUserHook@20
                  _VDDQueryDMA@12
                  _VDDReleaseDosHandle@8
                  _VDDReleaseIrqLine@8
                  _VDDRequestDMA@16
                  _VDDReserveIrqLine@8
                  _VDDRetrieveNtHandle@16
                  _VDDSetDMA@16
                  _VDDSimulate16@0
                  _VDDTerminateVDM@0
                  _VdmDbgAttach@0
                  _VdmGetParametersInfoError@0
                  _VdmMapFlat@12
                  _VdmParametersInfo@12
                  _VdmTraceEvent@12
                  _WOWSysErrorBox@20
                  _WaitIfIdle@0
                  _call_ica_hw_interrupt@12
                  _cmdCheckTemp@4
                  _cmdCheckTempInit@0
                  _cpu_createthread@4
                  _demClientErrorEx@12
                  _demFileDelete@4
                  _demFileFindFirst@12
                  _demFileFindNext@4
                  _demGetCurrentDirectoryLCDS@8
                  _demGetFileTimeByHandle_WOW@4
                  _demIsShortPathName@8
                  _demLFNCleanup@0
                  _demLFNGetCurrentDirectory@8
                  _demSetCurrentDirectoryGetDrive@8
                  _demSetCurrentDirectoryLCDS@8
                  _demWOWLFNAllocateSearchHandle@4
                  _demWOWLFNCloseSearchHandle@4
                  _demWOWLFNEntry@4
                  _demWOWLFNGetSearchHandle@4
                  _demWOWLFNInit@4
                  _fSeparateWow
                  _getAF@0
                  _getAH@0
                  _getAL@0
                  _getAX@0
                  _getBH@0
                  _getBL@0
                  _getBP@0
                  _getBX@0
                  _getCF@0
                  _getCH@0
                  _getCL@0
                  _getCS@0
                  _getCX@0
                  _getDF@0
                  _getDH@0
                  _getDI@0
                  _getDL@0
                  _getDS@0
                  _getDX@0
                  _getEAX@0
                  _getEBP@0
                  _getEBX@0
                  _getECX@0
                  _getEDI@0
                  _getEDX@0
                  _getEFLAGS@0
                  _getEIP@0
                  _getES@0
                  _getESI@0
                  _getESP@0
                  _getFS@0
                  _getGS@0
                  _getIF@0
                  _getIP@0
                  _getIntelRegistersPointer@0
                  _getMSW@0
                  _getOF@0
                  _getPF@0
                  _getSF@0
                  _getSI@0
                  _getSP@0
                  _getSS@0
                  _getZF@0
                  _host_CreateThread@24
                  _host_ExitThread@4
                  _host_com_close@4
                  _host_direct_access_error@4
                  _host_simulate@0
                  _pDeviceChain
                  _setAF@4
                  _setAH@4
                  _setAL@4
                  _setAX@4
                  _setBH@4
                  _setBL@4
                  _setBP@4
                  _setBX@4
                  _setCF@4
                  _setCH@4
                  _setCL@4
                  _setCS@4
                  _setCX@4
                  _setDF@4
                  _setDH@4
                  _setDI@4
                  _setDL@4
                  _setDS@4
                  _setDX@4
                  _setEAX@4
                  _setEBP@4
                  _setEBX@4
                  _setECX@4
                  _setEDI@4
                  _setEDX@4
                  _setEFLAGS@4
                  _setEIP@4
                  _setES@4
                  _setESI@4
                  _setESP@4
                  _setFS@4
                  _setGS@4
                  _setIF@4
                  _setIP@4
                  _setMSW@4
                  _setOF@4
                  _setPF@4
                  _setSF@4
                  _setSI@4
                  _setSP@4
                  _setSS@4
                  _setZF@4

  Summary

          A8 .debug$S
          14 .idata$2
          14 .idata$3
           4 .idata$4
           4 .idata$5
           A .idata$6


Regards,  Mike

[jj2007]

BOP is voodoo, nobody here is old enough for that 8)

(but the reactos guys have something here...)

[hutch--]

I would be inclined to use LoadLibrary() and GetProcAddress() to test the function calls you are after.

[pcMike]

I would be inclined to use LoadLibrary() and GetProcAddress() to test the function calls you are after.

But the issue is occurring before any (known) Win32 function calls are made. It's something at the BOP level which is failing to accept the DLL as being valid. I can't call LoadLibrary() from the DOS stub.

Regards,  Mike

[nidud]

deleted

[hutch--]

Normally in Win32 a DOS stub serves the purpose of advising a DOS user that the executable is for a later OS version and as 16 bit real mode code, it would not have access to a 32 bit PE DLL. I would have a look at the guts of the library you have that works to see what format it is in but I seriously doubt its a Microsoft /COFF format library, it sounds like a DOS style driver.

[pcMike]

Hutch,

In this case the DOS "stub" uses the BOP instructions to register a Win32 DLL and then call it's functions and later on unregister the DLL.
In my case the stub is a DOS TSR that uses interrupts to communicate with the Win32 DLL. I attached the old working version of the NTVDM.LIB library in my initial post.

This article describes the DOS BOP instruction set:
http://www.ragestorm.net/tutorial?id=27

Regards.  Mike

[hutch--]

Just having a quick read of the link you provided, you would be restricted to XP or earlier and relying on an undocumented capacity to interface 16 bit DOS with Win32. Now again if you need to do stuff like this and you have a working library, I would try and disassemble the library to find out what format it is written in. You may them be able to reproduce a library that performs tasks of that type with whatever functionality you have in mind. This is not a bug in the include and lib file, it is an undocumented hack to interface 16 bit code with Win32 code.

[TouEnMasm]

The MGetVdmPointer function had a limited usage,i386.
Here an extract of VDDSVC.sdk.
Have you an i386 computer ????????
seems you have no other choice than use another function.

IFDEF i386
Sim32FlushVDMPointer   equ   < r( addess, size, buffer, mode)>
MGetVdmPointer PROTO :DWORD ,:DWORD ,:DWORD
ELSE
Sim32GetVDMPointer PROTO :DWORD ,:DWORD ,:DWORD
Sim32FlushVDMPointer PROTO :DWORD ,:DWORD ,:XMASM ,:DWORD
ENDIF

[pcMike]

Hutch,  This is a program which I originally wrote in 2001, and it works perfectly under 32-bit Windows 10, 8.1, 8, 7, XP, 2000 as well as every 32-bit version of Windows Server (even Core).

It only fails if I substitute my old NTVDM.LIB with the one you included in MASM32 v10 or v11. In previous versions of MASM32, you did not include the NTVDM.LIB.

ToutEnMasm: It works on my Intel Xeon, I7, I5, I3, Core 2 duo, P4, as well as AMD CPU's. It works on both physical machines, and virtual environments including VMware and HyperV. The DLL uses the following settings and includes:
    .686
    .model flat, stdcall
    option casemap :none
    include \masm32\include\windows.inc
    include \masm32\include\kernel32.inc
    include \masm32\include\ntvdm.inc
    includelib \masm32\lib\kernel32.lib
    includelib \masm32\lib\ntvdm.lib

The DLL uses pipes to redirect data to an EXE Winsock Communication engine, effectively allowing DOS applications such as BBS software to be accessed via telnet.  (For some reason, I could not use Winsock directly from the DLL itself).

Regards,  Mike

[pcMike]

I should mention that BOP functions are not undocumented Voodoo. They are documented in older versions of the Microsoft DDK.
https://archive.org/stream/WinNT3.1DDKDoc/Win32SubSysGuide#page/n245/

There is also more information here:
http://sta.c64.org/blog/dosvddaccess.html

Regards,  Mike