a .dll with a function that opens a modal window like GETOPENFILENAME does

[fletbri]

How can I create a .dll containing functions similar to GetOpenFileName does in comdlg32.lib

so that when the user calls my .dll function, a modal window (or dialog) opens and the user interacts with it (much similar to an application).  they hit an exit Button to close the window and the function does its processing of data the user has entered.

I don't even know where to start.  Do I need WinMain() in my DLL to make the window's processing work?

[NoCforMe]

A DLL can contain any kinds of functions.

Creating a function like GetOpenFileName() is not a trivial exercise (but it is do-able). The main problem is going to be getting, displaying and navigating to all those folders and files. Current Win32 practice would involve using "shell" functions, which I have no experience using.

WinMain() has nothing at all to do with DLLs. You simply code up your functions and create the library. There'a a little bookkeeping you need to do when creating the DLL, but it's not difficult. Maybe others here can detail the process; I've only done it once myself (successfully).

[fletbri]

  My aim is not to write a replacement to the GETOPENFILENAME function, but rather to create a function which at first opens a window (or modal dialog) that the caller interacts with, enters data, checks an option in a checkbox, clicks a button to start a process, or clicks a button to close the modal window.  The only similarity to GETOPENFILENAME is that GETOPENFILENAME opens a modal dialog and processes window messages from the user interactions. 

The reason I mention GETOPENFILENAME is that it proves that what I am planning to do is possible.

[NoCforMe]

OK, good to have more info on what you're trying to do.

That can certainly go into a DLL. So far as the dialog goes, you'll need to create either a dialog resource or a dialog template. What's the difference?

A dialog resource (created by a resource editor, natch) defines the dialog, including all its child controls, in a resource that gets compiled (by a resource compiler, natch) and embedded in the DLL. A template, on the other hand, is just an in-memory structure that you create which you can then pass to the dialog function (DialogBoxIndirectParam() for a modal dialog) to start the dialog.

I haven't done either of these; the one DLL I created does create a window which contains some child controls, so it's somewhat like a dialog. I don't know how you'd embed a resource in a DLL (others here can help with this), but I imagine it might be easier to use a dialog template. Thing is, how do you create one? I have a program I wrote that I use to create templates, basically a dialog editor that creates a template instead of an embedded resource. But it's kinda buggy ...

Anyhow, what you're proposing is definitely do-able. The thing to do would be to first code it as just a function within a Win32 program to test it, then convert it to a DLL.

BTW, instead of WinMain(), a DLL has a DLLmain() entry point. (Not sure if this is a "magic name" or if it gets specified when creating the DLL.)

[fearless]

Couple of useful links related to this subject:

https://www.masmforum.com/board/index.php?topic=3780.0

https://www.masmforum.com/board/index.php?topic=6442.0

Just have to remember to use hInstance of the dll when loading any resources stored in the dll - so save the hInstDLL parameter from DllMain (https://learn.microsoft.com/en-us/windows/win32/dlls/dllmain) to some variable, hDLLInstance for example.

[NoCforMe]

Code Select Expand

;====================================
; DLL main entry point proc
;====================================

DLLmain        PROC hInstDLL:HINSTANCE, reason:DWORD, reserved:DWORD

; Store instance handle where we can get at it:
    MOV    EAX, hInstDLL
    MOV    InstanceHandle, EAX

    MOV    EAX, TRUE
    RET

DLLmain        ENDP
I'm not sure what those other 2 parameters ("reason" and "reserved") are, but I only needed the instance handle,

[NoCforMe]

Here's what a dialog memory template looks like. This is a small dialog with a static control, an edit control and two buttons:

Code Select Expand

;********  Memory Dialog Template for "EdAsm editor" Project ********
;********  Created by DialogGen on 7/28/2023 at 9:15 pm  ********

$lineNumEdit EQU 1201

;====== Dialog header: ======
DD WS_BORDER or WS_CLIPCHILDREN or WS_CLIPSIBLINGS or WS_DLGFRAME or WS_POPUP or WS_SYSMENU or WS_VISIBLE or DS_SETFONT
DD WS_EX_CLIENTEDGE
DW 4 ;# items
DW 22 ;X
DW 5 ;Y
DW 105 ;Width
DW 46 ;Height
DW 0 ;Menu ID
DW 0 ;Class array
DB 'G',0, 'o',0, ' ',0, 'T',0, 'o',0, ' ',0, 'L',0, 'i',0, 'n',0, 'e',0
DW 0 ;Unicode terminator
DW 9 ;Font size
DB 'S',0, 'e',0, 'g',0, 'o',0, 'e',0, ' ',0, 'U',0, 'I',0
DW 0 ;Unicode terminator

;====== Dialog item (Static): ======
DD WS_CHILD or WS_VISIBLE
DD 0
DW 24 ;X
DW 10 ;Y
DW 22 ;Width
DW 8 ;Height
DW 100 ;Control ID
DW 0FFFFh ;Std. ctl. ID
DW 82h ;Std. ctl. type
DB 'L',0, 'i',0, 'n',0, 'e',0, ' ',0, '#',0, ':',0
DW 0 ;Unicode terminator
DW 0 ;Creation data

;====== Dialog item (Edit): ======
DD WS_BORDER or WS_CHILD or WS_TABSTOP or WS_VISIBLE
DD 0
DW 47 ;X
DW 9 ;Y
DW 28 ;Width
DW 10 ;Height
DW 1201 ;Control ID
DW 0FFFFh ;Std. ctl. ID
DW 81h ;Std. ctl. type
DW 0 ;Unicode terminator
DW 0 ;Creation data
DW 0 ;align to DWORD

;====== Dialog item (Button): ======
DD BS_CENTER or WS_CHILD or WS_TABSTOP or WS_VISIBLE
DD 0
DW 10 ;X
DW 28 ;Y
DW 34 ;Width
DW 12 ;Height
DW IDOK ;Control ID
DW 0FFFFh ;Std. ctl. ID
DW 80h ;Std. ctl. type
DB 'O',0, 'K',0
DW 0 ;Unicode terminator
DW 0 ;Creation data
DW 0 ;align to DWORD

;====== Dialog item (Button): ======
DD BS_CENTER or WS_CHILD or WS_TABSTOP or WS_VISIBLE
DD 0
DW 60 ;X
DW 28 ;Y
DW 34 ;Width
DW 12 ;Height
DW IDCANCEL ;Control ID
DW 0FFFFh ;Std. ctl. ID
DW 80h ;Std. ctl. type
DB 'C',0, 'a',0, 'n',0, 'c',0, 'e',0, 'l',0
DW 0 ;Unicode terminator
DW 0 ;Creation data


[NoCforMe]

Hey, I found my "recipe" for making a DLL:

===========================
  Tasty Recipe for DLLs
===========================

DLLmain procedure
DLLmain    PROC    hInstDLL:HINSTANCE, reason:DWORD, reserved:DWORD

o Be sure to save instance handle globally.
o Return TRUE to make sure DLL gets loaded
o Set module's entry point to this proc:
    END DLLmain
(So "DLLmain" is NOT a magic name: whatever procedure is at this location will be called when the DLL is loaded.)

Assembling
Assemble normally:
    ml /c /coff /Fl /Sg /Sn <.asm file>

Export DEF file
LIBRARY <DLL name>
EXPORTS <procedure name>
EXPORTS <procedure name>
    ...

Linking
    link /SUBSYSTEM:WINDOWS /DLL /DEF:<.def file> <.obj file>

Bake at 350° for 10 minutes ...

[TimoVJL]

link.exe and polink.exe knows these native startup function names:

Code Select Expand

x86
BOOL WINAPI _DllMainCRTStartup(HANDLE hInstDLL, DWORD dwReason, LPVOID lpvReserved)
x64
BOOL WINAPI DllMainCRTStartup(HANDLE hInstDLL, DWORD dwReason, LPVOID lpvReserved)
DLLs and Visual C++ run-time library behavior

[NoCforMe]

I went ahead and whipped up a little demo. Which proves that what the OP asked for is not only possible but easily done.

Demo program (DLL with dialog demo.exe) uses a DLL (DLLtest.dll) to  put up a dialog. To prove it works, try running the demo without the DLL, then "install" the DLL (put it in the same folder as the demo) and try it.

In order not to do all your homework for you, I'm leaving the implementation details to you (the OP). My recipe above will get you there. Oh, and you can definitely embed a dialog as a resource (or any other resource) in the DLL; just add it to the list of object modules when you link it.