The MASM Forum

General => The Campus => Topic started by: Matthias Merkel on December 17, 2017, 11:00:20 PM

Title: Download a file from the internet
Post by: Matthias Merkel on December 17, 2017, 11:00:20 PM
I've been trying to figure this out for a few hours now but wasn't able to find a way to download a file from the internet. Does anyone know how that would be possible to do?
Title: Re: Download a file from the internet
Post by: jj2007 on December 17, 2017, 11:37:38 PM
Like this?

include \masm32\MasmBasic\MasmBasic.inc         ; download (http://masm32.com/board/index.php?topic=94.0)
  Init
  FileWrite "test.html", FileRead$("http://masm32.com/board/index.php?board=1.0")
  ShEx "test.html"   ; show it in your browser
EndOfCode


Title: Re: Download a file from the internet
Post by: Matthias Merkel on December 17, 2017, 11:52:27 PM
That is awesome! Thank you so much.
Title: Re: Download a file from the internet
Post by: aw27 on December 18, 2017, 01:12:10 AM
 :biggrin:

You can also do it in pure VB.Net and have lots of fun too:


        Using client As System.Net.WebClient = New System.Net.WebClient()
            client.DownloadFile("http://masm32.com/board/index.php?board=1.0", "test.html")
        End Using

Title: Re: Download a file from the internet
Post by: Matthias Merkel on December 18, 2017, 03:36:46 AM
Quote from: aw27 on December 18, 2017, 01:12:10 AM
:biggrin:

You can also do it in pure VB.Net and have lots of fun too:


        Using client As System.Net.WebClient = New System.Net.WebClient()
            client.DownloadFile("http://masm32.com/board/index.php?board=1.0", "test.html")
        End Using


The question was about ASM though.
Title: Re: Download a file from the internet
Post by: aw27 on December 18, 2017, 03:56:55 AM
The important thing is that you are happy with what you have learnt about ASM.
Title: Re: Download a file from the internet
Post by: felipe on December 18, 2017, 04:16:35 AM
 :P

Welcome to the forum Matthias Merkel.   :icon14:
Title: Re: Download a file from the internet
Post by: Matthias Merkel on December 18, 2017, 04:22:30 AM
Quote from: felipe on December 18, 2017, 04:16:35 AM
:P

Welcome to the forum Matthias Merkel.   :icon14:

Thanks
Title: Re: Download a file from the internet
Post by: Vortex on December 18, 2017, 06:55:05 AM
Hi Matthias Merkel,

Here is a quick example :

.386
.model flat,stdcall
option casemap:none

include     \masm32\include\windows.inc
include     \masm32\include\kernel32.inc
include     \masm32\include\urlmon.inc

includelib  \masm32\lib\kernel32.lib
includelib  \masm32\lib\urlmon.lib

.data

szURL       db 'http://vortex.masmcode.com/files/Wcrt0_7.zip',0
FileName    db 'TinyCRT.zip',0

.code

start:

    invoke  URLDownloadToFile,0,ADDR szURL,\
            ADDR FileName,0,0

    invoke  ExitProcess,0

END start


The trick is to use the API function URLDownloadToFile.
Title: Re: Download a file from the internet
Post by: hutch-- on December 18, 2017, 12:16:20 PM
I wonder if anyone knows how to construct the callback for the last argument to get download progress. The API works fine in 64 bit but it has a COM interface for the callback that I have no data on at all. Other than that, is there any viable alternatives for win64 (win7 up) that can both do the download AND have progress information ?

This in 64 bit works fine.

invoke URLDownloadToFile,0,url,targ,0,0       ; pCB
Title: Re: Download a file from the internet
Post by: Matthias Merkel on December 18, 2017, 04:49:33 PM
Quote from: Vortex on December 18, 2017, 06:55:05 AM
Hi Matthias Merkel,

Here is a quick example :

.386
.model flat,stdcall
option casemap:none

include     \masm32\include\windows.inc
include     \masm32\include\kernel32.inc
include     \masm32\include\urlmon.inc

includelib  \masm32\lib\kernel32.lib
includelib  \masm32\lib\urlmon.lib

.data

szURL       db 'http://vortex.masmcode.com/files/Wcrt0_7.zip',0
FileName    db 'TinyCRT.zip',0

.code

start:

    invoke  URLDownloadToFile,0,ADDR szURL,\
            ADDR FileName,0,0

    invoke  ExitProcess,0

END start


The trick is to use the API function URLDownloadToFile.

The huge issue with that way is that it's detected by a lot of antivirus vendors as malware: https://www.virustotal.com/#/file/4785d45664ef436c1f2cbd11e9dd46a358aee36f323fd213101c289b5632b523/detection
Title: Re: Download a file from the internet
Post by: aw27 on December 18, 2017, 06:12:03 PM
Quote from: hutch-- on December 18, 2017, 12:16:20 PM
Other than that, is there any viable alternatives for win64 (win7 up) that can both do the download AND have progress information ?
I prefer InternetOpenUrl, HttpQueryInfo, InternetReadFile. HttpQueryInfo gives you the number of bytes to read.
Title: Re: Download a file from the internet
Post by: aw27 on December 18, 2017, 06:14:15 PM
Quote
The huge issue with that way is that it's detected by a lot of antivirus vendors as malware
The antivirus are particularly dumb and many of them hate polink.
Title: Re: Download a file from the internet
Post by: jj2007 on December 18, 2017, 07:21:25 PM
Quote from: Matthias Merkel on December 18, 2017, 04:49:33 PMThe huge issue with that way is that it's detected by a lot of antivirus vendors as malware

Yes indeed. Try the attached version, btw with progress indicator in the statusbar (Jotti:  2/18 scanners reported malware (https://virusscan.jotti.org/en-US/filescanjob/2kb2t7q8qs)). Here is the source:GuiParas equ "Downloading a file is easy"
include \masm32\MasmBasic\Res\MbGui.asm
GuiControl Edit, "edit"
GuiControl Status, "statusbar"
Download "http://masm32.com/board/index.php?topic=6771.msg72451#msg72451", cb:MyCallback, msg:hWnd_
Event Message
  .if uMsg==WM_DOWNLOADFINISHED
SetWin$ hEdit="Download finished"+CrLf$+NoTag$(wParam)
MsgBox 0, "Save contents to test.html?", "Hi", MB_YESNOCANCEL
If_ eax==IDYES Then <mcs FileWrite "test.html", wParam : ShEx "test.html">
  .endif
EndOfEvents
  MyCallback:
  SetWin$ hStatus=Str$("%i bytes\ndownloaded - hold Shift to stop it", edx) ; currrent state in edx
  invoke GetKeyState, VK_SHIFT
  test ah, ah ; stop download if Sign? flag set
  retn
GuiEnd
Title: Re: Download a file from the internet
Post by: avcaballero on December 18, 2017, 09:19:13 PM
Quote from: jj2007 on December 18, 2017, 07:21:25 PM
Quote from: Matthias Merkel on December 18, 2017, 04:49:33 PMThe huge issue with that way is that it's detected by a lot of antivirus vendors as malware

Yes indeed. Try the attached version, btw with progress indicator in the statusbar (Jotti:  2/18 scanners reported malware (https://virusscan.jotti.org/en-US/filescanjob/2kb2t7q8qs)). Here is the source:GuiParas equ "Downloading a file is easy"
include \masm32\MasmBasic\Res\MbGui.asm
GuiControl Edit, "edit"
GuiControl Status, "statusbar"
Download "http://masm32.com/board/index.php?topic=6771.msg72451#msg72451", cb:MyCallback, msg:hWnd_
Event Message
  .if uMsg==WM_DOWNLOADFINISHED
SetWin$ hEdit="Download finished"+CrLf$+NoTag$(wParam)
MsgBox 0, "Save contents to test.html?", "Hi", MB_YESNOCANCEL
If_ eax==IDYES Then <mcs FileWrite "test.html", wParam : ShEx "test.html">
  .endif
EndOfEvents
  MyCallback:
  SetWin$ hStatus=Str$("%i bytes\ndownloaded - hold Shift to stop it", edx) ; currrent state in edx
  invoke GetKeyState, VK_SHIFT
  test ah, ah ; stop download if Sign? flag set
  retn
GuiEnd

Wow, it is a quite good example, where are the masm sources?  :biggrin:
Title: Re: Download a file from the internet
Post by: aw27 on December 18, 2017, 09:27:57 PM
To prevent issues with antivirus (https://virusscan.jotti.org/en-US/filescanjob/kzzy27svtq) I will start programming in pure VB.Net  in 2018 :badgrin:
It produces small executables too.


Imports System.Net

Module Module1
    Public WithEvents client As New WebClient()
    Public stillDownloading As Boolean

    Sub Main()
        stillDownloading = True
        client.DownloadFileAsync(New Uri("http://masm32.com/board/index.php?topic=6771.0"), "test.html")
        While stillDownloading
        End While

        Dim filePath As String = ".\test.html"
        Dim psi As New ProcessStartInfo(filePath)
        psi.WorkingDirectory = IO.Path.GetDirectoryName(filePath)
        Process.Start(psi)
    End Sub

    Private Sub Client_DownloadProgressChanged(ByVal sender As Object, ByVal e As DownloadProgressChangedEventArgs) Handles client.DownloadProgressChanged
        Console.WriteLine(String.Format("{0} %", e.ProgressPercentage))
        If e.ProgressPercentage = 100 Then
            Console.WriteLine("Download completed")
            stillDownloading = False
        End If
    End Sub


To be more confident let me check in another site:
https://www.virustotal.com/#/file/283bf0921aaffd360edec353d9698e679e6dfaf12319100c7d2451a42f705899/detection
Damn, CrowdStrike Falcon (never heard about it before) says it is malware with 80% confidence.   :(

Title: Re: Download a file from the internet
Post by: jj2007 on December 18, 2017, 09:41:05 PM
Quote from: caballero on December 18, 2017, 09:19:13 PM
Wow, it is a quite good example, where are the masm sources?  :biggrin:

Thanks :t

This is a MASM source (it assembles with Masm, right?). If you need more detail, check \Masm32\MasmBasic\MasmBasic.inc and \Masm32\MasmBasic\Res\MbGui.asm
Title: Re: Download a file from the internet
Post by: jj2007 on December 18, 2017, 10:03:23 PM
Quote from: aw27 on December 18, 2017, 09:27:57 PMI will start programming in pure VB.Net  in 2018 :badgrin:

These are good intentions, José :t

Your code has little problems, though:TmpFile.bas(14) : error BC30002: Tipo 'ProcessStartInfo' non definito
TmpFile.bas(16) : error BC30451: 'Process' non dichiarato

(warning: VB is a Micros**t product!)
Title: Re: Download a file from the internet
Post by: aw27 on December 18, 2017, 10:49:43 PM
Quote
[quote author=jj2007 link=topic=6771.msg72479#msg72479 date=1513595003]
TmpFile.bas(16) : error BC30451: 'Process' non dichiarato

(warning: VB is a Micros**t product!)
It works here, I was even able to run it, imagine!
Title: Re: Download a file from the internet
Post by: aw27 on December 18, 2017, 10:52:19 PM
Quote from: jj2007 on December 18, 2017, 09:41:05 PM
This is a MASM source (it assembles with Masm, right?). If you need more detail, check \Masm32\MasmBasic\MasmBasic.inc and \Masm32\MasmBasic\Res\MbGui.asm
It was built with MASM (probably) and it builds with MASM because you had not the time (yet) to build your own HLL compiler. May be in 2018 you can handle that part.
Title: Re: Download a file from the internet
Post by: jj2007 on December 18, 2017, 11:16:32 PM
Quote from: aw27 on December 18, 2017, 10:49:43 PMIt works here, I was even able to run it, imagine!

After searching a bit for the error messages, I found the culprit:
Imports System.Net ' José's original code
Imports System.Diagnostics ' added by JJ to make it work
...
End Module ' added by JJ to make it work


And voilà, it builds fine, and even starts downloading something :t
Then it gets stuck and hangs at 100% cpu load. Fortunately, it can be killed with Task Manager :icon_mrgreen:
Title: Re: Download a file from the internet
Post by: mabdelouahab on December 19, 2017, 01:34:42 AM
with : VB.Net Masm64.Net  :biggrin:

;*************************************************************
OPTION LITERALS:ON
include \masm32\include64\AsmDotNet64.inc
   .includelib kernel32,msvcrt,user32
   .proto ExitProcess,wprintf,printf,MessageBoxA
;*************************************************************
   .import System.Net.WebClient,tWebClient
   .import System.Net.DownloadProgressChangedEventArgs,tDPCEventArgs
   .importSystem.Uri,tUri
   .import System.Diagnostics.ProcessStartInfo,tProcessStartInfo
   .import System.Diagnostics.Process,tProcess
   .import System.IO.Path,tPath
;*************************************************************
.data
   stillDownloading    dd 1
   filePath          dq Of_Bstr(".\test.html")
.code
;*************************************************************
   On_DownloadProgressChanged proc _Sender:QWORD,_e:QWORD
   .LOCAL e,tDPCEventArgs
   LOCAL ProgressPercentage:QWORD
      mov rax,_e
      mov e,rax
      .get ,e.ProgressPercentage
      mov ProgressPercentage,rax
      invoke printf,"\n{%d}",ProgressPercentage
      .if ProgressPercentage == 100
         mov stillDownloading,0
      .endif
      ret
   On_DownloadProgressChanged endp
;*************************************************************
   entry_point proc
      .LOCAL client,tWebClient
      .LOCAL vUri,tUri
      .LOCAL psi,tProcessStartInfo
      .LOCAL Path,tPath
      .LOCAL Process,tProcess
      LOCAL DirectoryName:qword
      .invoke vUri.CreateInstance,"http://masm32.com/board/index.php?topic=6771.0"
      .invoke client.CreateInstance
      .invoke client.DownloadFileAsync,vUri,"test.html"
      .addhandlerEx client.DownloadProgressChanged, addr On_DownloadProgressChanged
   @@:   cmp stillDownloading,1
      je @B
      .invoke psi.CreateInstance,filePath
      .invoke Path.GetDirectoryName,filePath
      mov DirectoryName,rax
      .set psi.WorkingDirectory ,DirectoryName
      .invoke Process.Start_5,psi
      invoke ExitProcess,0
      ret
   entry_point endp
;*************************************************************
end    
;*************************************************************
Title: Re: Download a file from the internet
Post by: aw27 on December 19, 2017, 02:00:30 AM
Quote
Then it gets stuck and hangs at 100% cpu load. Fortunately, it can be killed with Task Manager :icon_mrgreen:
@JJ
1- Imports System.Diagnostics is a default imported namespace in my VS 2017 installation and I swear that it is the first time I use VB.Net in 2017 and I have not changed anything.

2. If it gets stuck is because you are trying to make a Forms Application and my example is for a Console application.
       While stillDownloading
        End While
will not make a console app get stuck because Client_DownloadProgressChanged comes from a different thread and the download is asynch.

 
Title: Re: Download a file from the internet
Post by: jj2007 on December 19, 2017, 03:27:31 AM
Quote from: aw27 on December 19, 2017, 02:00:30 AM
Quote
Then it gets stuck and hangs at 100% cpu load. Fortunately, it can be killed with Task Manager :icon_mrgreen:
@JJ
1- Imports System.Diagnostics is a default imported namespace in my VS 2017 installation and I swear that it is the first time I use VB.Net in 2017 and I have not changed anything.

Me swear, too, that I have not changed anything :biggrin:

Quote2. If it gets stuck is because you are trying to make a Forms Application and my example is for a Console application.
       While stillDownloading
        End While
will not make a console app get stuck because Client_DownloadProgressChanged comes from a different thread and the download is asynch.

Console output:
0 %
0 %
100 %
Download completed


Doesn't look like a "Forms Application", whatever that is 8)

Btw that doesn't look like good programming practice. Shouldn't there be at least a Sleep(1) inside?        While stillDownloading
        End While
Title: Re: Download a file from the internet
Post by: aw27 on December 19, 2017, 04:04:13 AM
Quote
Btw that doesn't look like good programming practice. Shouldn't there be at least a Sleep(1) inside?
Yeah, but I could not remember how to say Sleep in VBNet-ish.  :lol:

Title: Re: Download a file from the internet
Post by: jj2007 on December 19, 2017, 04:10:32 AM
Quote from: aw27 on December 19, 2017, 04:04:13 AM
Quote
Btw that doesn't look like good programming practice. Shouldn't there be at least a Sleep(1) inside?
Yeah, but I could not remember how to say Sleep in VBNet-ish.  :lol:

The learning curve will be steep, but you are still young, so we are all optimistic :icon14:
Quote from: aw27 on December 18, 2017, 09:27:57 PMI will start programming in pure VB.Net  in 2018 :badgrin:

P.S.: Just tested my download snippet (http://masm32.com/board/index.php?topic=6771.msg72475#msg72475) in Win10 home, it builds & works like a charm. Normal user account, it doesn't even warn me - I am always impressed how easy it is to download stuff in Windows. No wonder people catch viruses :badgrin:
Title: Re: Download a file from the internet
Post by: aw27 on December 19, 2017, 04:29:59 AM
Quote
The learning curve will be steep
Every kid can learn VB Net in a few hours.  :icon14:
I like to know a bit of every programming language, but am not into those that try to do everything for me.
BTW, I found that Sleep is System.Threading.Thread.CurrentThread.Sleep
On the other end you can hang a Forms application with long sleeps. What I use to do in other programming languages is a small sleep followed by an instruction to pump the waiting messages. This is probably doEvents in VB Net.
Title: Re: Download a file from the internet
Post by: juozas on December 29, 2017, 05:42:15 AM
A long time I created an example program that Downloads and saves a file from url you input in a dialog it shows. It remembers the last url you downloaded:
.386
.model flat, stdcall  ;32 bit memory model
option casemap :none  ;case sensitive

include \masm32\include\windows.inc

uselib MACRO libname
include \masm32\include\libname.inc
includelib \masm32\lib\libname.lib
endm

uselib kernel32
uselib user32
uselib shlwapi
uselib Comctl32
uselib wininet
uselib masm32

include \masm32\macros\macros.asm

DlgProc PROTO :HWND,:UINT,:WPARAM,:LPARAM
GetTheFile PROTO :DWORD

.const

IDD_MAIN                equ 101
IDC_OK                  equ 1
IDC_CANCEL              equ 2
IDC_FNAME               equ 102
IDI_ICON                equ 101


.data
msgOk db "Download is done!", 0
msgErr db "Download failed", 0
mcap db "Download",0

.data?

FileName db 256 dup (?)

hInstance dd ?
hIcon dd ?
hCfg dd ?
CfgSize dd ?
CfgRead dd ?
CfgWrite dd ?

.code

start:
invoke GetModuleHandle,NULL
mov hInstance,eax
invoke InitCommonControls
invoke DialogBoxParam,hInstance,IDD_MAIN,NULL,addr DlgProc,NULL
invoke ExitProcess,0

DlgProc proc hWin:HWND,uMsg:UINT,wParam:WPARAM,lParam:LPARAM
.if uMsg==WM_INITDIALOG
;initialization here
invoke LoadIcon, hInstance, IDI_ICON
        mov hIcon, eax
        invoke SendMessage, hWin, WM_SETICON, 1, hIcon
        invoke CreateFile,chr$("DownDlg.cfg"),GENERIC_READ or GENERIC_WRITE,FILE_SHARE_READ or FILE_SHARE_WRITE,NULL,OPEN_EXISTING,FILE_ATTRIBUTE_ARCHIVE,NULL
        mov hCfg, eax
        invoke GetFileSize,hCfg,NULL
        mov CfgSize, eax
        invoke ReadFile,hCfg,addr FileName, CfgSize, addr CfgRead, NULL
        invoke CloseHandle, hCfg
        invoke SetDlgItemText, hWin, IDC_FNAME, addr FileName
        invoke GetDlgItem,hWin,IDC_FNAME
      invoke SetFocus,eax
.elseif uMsg==WM_COMMAND
.if wParam==IDC_OK
invoke GetDlgItemText,hWin,IDC_FNAME,addr FileName,sizeof FileName
invoke GetTheFile, addr FileName
.if eax == TRUE
invoke CreateFile,chr$("DownDlg.cfg"),GENERIC_READ or GENERIC_WRITE,FILE_SHARE_READ or FILE_SHARE_WRITE,NULL,CREATE_ALWAYS,FILE_ATTRIBUTE_ARCHIVE,NULL
mov hCfg, eax

invoke WriteFile,hCfg,addr FileName,sizeof FileName, addr CfgWrite,NULL
invoke SetEndOfFile,hCfg
invoke CloseHandle,hCfg
invoke MessageBox, hWin, addr msgOk, addr mcap, MB_OK + MB_ICONINFORMATION
invoke SendMessage,hWin,WM_CLOSE,NULL,NULL
.else
    invoke MessageBox, hWin, addr msgErr, addr mcap, MB_OK + MB_ICONSTOP
invoke GetDlgItem,hWin,IDC_FNAME
      invoke SetFocus,eax
.endif
.elseif wParam==IDC_CANCEL
invoke SendMessage,hWin,WM_CLOSE,NULL,NULL
.endif
.elseif uMsg==WM_CLOSE
invoke EndDialog,hWin,0
.endif
xor eax,eax
ret
DlgProc endp

GetTheFile proc fName:DWORD
local buffer[1024]: BYTE
local hSession: DWORD
local hUrl: DWORD
local bufferlen: DWORD

local AppName[127]: BYTE
local FileSave[MAX_PATH]: BYTE

local fhandl: DWORD
local bwrite: DWORD

invoke PathFindFileName, fName
invoke lstrcpy, addr FileSave, eax

invoke GetModuleFileName,  NULL,  addr AppName,  sizeof AppName
invoke InternetOpen,  addr AppName,  INTERNET_OPEN_TYPE_PRECONFIG, NULL,  NULL,  NULL
mov hSession, eax
.if hSession == INVALID_HANDLE_VALUE
    mov eax,  FALSE
    ret
    .endif
invoke InternetOpenUrl, hSession, fName, NULL, NULL, NULL, NULL
mov hUrl, eax
.if hUrl == INVALID_HANDLE_VALUE
    mov eax, FALSE
    ret
.endif

invoke CreateFile,addr FileSave,  GENERIC_READ or GENERIC_WRITE,  FILE_SHARE_READ, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL
mov fhandl, eax
.if fhandl == INVALID_HANDLE_VALUE
mov eax, FALSE
ret
.endif

invoke SetFilePointer,  fhandl, NULL, NULL, FILE_BEGIN
download:
invoke InternetReadFile, hUrl,  addr buffer,sizeof buffer,  addr bufferlen
.if bufferlen != 0
invoke WriteFile,  fhandl,  addr buffer,  bufferlen,  addr bwrite, NULL
jmp download
.endif

invoke CloseHandle,  fhandl
invoke InternetCloseHandle,  hUrl
invoke InternetCloseHandle,  hSession
mov eax, TRUE
ret
GetTheFile endp

end start

It uses resources for dialog definition. Compiled under wine. Full archive attached.
Edit: It hangs if it can't resolve and/or connect to url (at least on wine), it must be full one (with the protocol prefix), it also will overwrite any existing file that's same as in url upon attempting to download it, so please be careful.