News:

Masm32 SDK description, downloads and other helpful links
Message to All Guests
NB: Posting URL's See here: Posted URL Change

Main Menu

FindFirstFileEx, FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH

Started by jj2007, May 28, 2013, 08:58:06 PM

Previous topic - Next topic
Print
Go Down Pages1 2
User actions

jj2007

May 28, 2013, 08:58:06 PM Last Edit: May 29, 2013, 08:25:09 PM by jj2007
I am trying to speed up file searches but no luck so far:

AMD Athlon(tm) Dual Core Processor 4450B (MMX, SSE, SSE2, SSE3)
Windows 7 Professional
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags

21369 µs for 100 loops, BasicInfo=1, LargeFetch=1
23.7.12 14:24:04        \Masm32\Examples\exampl01
20.2.12 18:15:47        \Masm32\Examples\enumerate

19708 µs for 100 loops, BasicInfo=0, LargeFetch=0
23.7.12 14:24:04        \Masm32\Examples\exampl01
20.2.12 18:15:47        \Masm32\Examples\enumerate

19834 µs for 100 loops, BasicInfo=1, LargeFetch=1
23.7.12 14:24:04        \Masm32\Examples\exampl01
20.2.12 18:15:47        \Masm32\Examples\enumerate

20297 µs for 100 loops, BasicInfo=0, LargeFetch=0
23.7.12 14:24:04        \Masm32\Examples\exampl01
20.2.12 18:15:47        \Masm32\Examples\enumerate

16150 µs for 100 loops, BasicInfo=1, LargeFetch=1
23.7.12 14:24:04        \Masm32\Examples\exampl01
20.2.12 18:15:47        \Masm32\Examples\enumerate

18467 µs for 100 loops, BasicInfo=0, LargeFetch=0
23.7.12 14:24:04        \Masm32\Examples\exampl01
20.2.12 18:15:47        \Masm32\Examples\enumerate

I've seen claims for miraculous improvements but my OS fails to honour these flags :(
Grateful for tests and ideas.

P.S.: I am aware of Lingo's stuff but it crashes. Probably too many GetProcAddress with ordinals ;)

dedndave

#1
May 28, 2013, 11:05:14 PM
Code Select
Intel(R) Pentium(R) 4 CPU 3.00GHz (MMX, SSE, SSE2, SSE3)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags

190 µs for 100 loops, BasicInfo=1, LargeFetch=1

3100 µs for 100 loops, BasicInfo=0, LargeFetch=0

251 µs for 100 loops, BasicInfo=1, LargeFetch=1

3045 µs for 100 loops, BasicInfo=0, LargeFetch=0

254 µs for 100 loops, BasicInfo=1, LargeFetch=1

3052 µs for 100 loops, BasicInfo=0, LargeFetch=0

jj2007

#2
May 29, 2013, 12:59:57 AM Last Edit: May 29, 2013, 05:29:55 PM by jj2007
Thanks, Dave.

This is really messy (MSDN FindFirstFileEx):
QuoteFIND_FIRST_EX_LARGE_FETCH .. and Windows XP:  This value is not supported until Windows Server 2008 R2 and Windows 7."

You just demonstrated the contrary :bgrin:
EDIT: No, that was just a bug in the first version.

Now why is it slow on my Win7-32 and -64 systems but fast on XP? ::)

To get the computer name, I added
        TryRTE RegX2, con
        Let edi=GetRegVal("HKLM\SYSTEM\CurrentControlSet\Control\ComputerName\ComputerName", "ComputerName")
        RegX2:
Works fine on W7-32 but requires admin privileges on W7-64; however, when I run it as admin, it does no longer read the network drive... oh my dear friends in Redmond...  :icon_mrgreen:

Gunther

#3
May 29, 2013, 03:58:44 AM
Jochen,

the results:

Code Select

Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SS
E4.2, AVX)
Windows 7 Professional
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags

1112 µs for 100 loops, BasicInfo=1, LargeFetch=1
01.01.1601      02:00:00
01.01.1601      02:00:00

804 µs for 100 loops, BasicInfo=0, LargeFetch=0
01.01.1601      02:00:00
01.01.1601      02:00:00

1042 µs for 100 loops, BasicInfo=1, LargeFetch=1
01.01.1601      02:00:00
01.01.1601      02:00:00

753 µs for 100 loops, BasicInfo=0, LargeFetch=0
01.01.1601      02:00:00
01.01.1601      02:00:00

1002 µs for 100 loops, BasicInfo=1, LargeFetch=1
01.01.1601      02:00:00
01.01.1601      02:00:00

919 µs for 100 loops, BasicInfo=0, LargeFetch=0
01.01.1601      02:00:00
01.01.1601      02:00:00
-- ok --


Gunther
You have to know the facts before you can distort them.

jj2007

#4
May 29, 2013, 05:46:50 AM
Thanxalot, Dave and Gunther. In the meantime, I fixed the bug that made it so fast on XP :icon_mrgreen:

   ; perform a dummy search to find out if the OS supports the fast flags:
   FindExInfoBasic=1
   invoke FindFirstFileEx, chr$("C:\NoFFx.Yet"),
   FindExInfoBasic, offset wfd, 0, 0, FIND_FIRST_EX_LARGE_FETCH
   invoke GetLastError <<<<<<<< I had forgotten this one :redface:
   .if eax!=ERROR_INVALID_PARAMETER
      add dword ptr [esi], FindExInfoBasic            ; ffxInfoLevel
      add dword ptr [esi+4], FIND_FIRST_EX_LARGE_FETCH   ; ffxAdditionalFlags
   .endif

New version attached at top of thread.

@Gunther: It searches \Masm32\Examples\* - since you had no results, that was probably the wrong drive or puter. The new version complains with a MsgBox if it doesn't find anything.

XP results:
Intel(R) Celeron(R) M CPU        420  @ 1.60GHz (MMX, SSE, SSE2, SSE3)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
MSI, 21:47:41

253 ms for 1000 loops, BasicInfo=0, LargeFetch=0
02.03.2013      00:51:53        \Masm32\Examples\exampl01
03.11.2012      12:16:41        \Masm32\Examples\Bill_Cravener

288 ms for 1000 loops, BasicInfo=0, LargeFetch=0
02.03.2013      00:51:53        \Masm32\Examples\exampl01
03.11.2012      12:16:41        \Masm32\Examples\Bill_Cravener

130 ms for 1000 loops, BasicInfo=0, LargeFetch=0
02.03.2013      00:51:53        \Masm32\Examples\exampl01
03.11.2012      12:16:41        \Masm32\Examples\Bill_Cravener

127 ms for 1000 loops, BasicInfo=0, LargeFetch=0
02.03.2013      00:51:53        \Masm32\Examples\exampl01
03.11.2012      12:16:41        \Masm32\Examples\Bill_Cravener

Flags are never set, i.e. it fails on XP as documented.

Gunther

#5
May 29, 2013, 07:14:48 AM
Jochen,

here are the results from my virtual machine:

Code Select

Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SS
E4.2, AVX)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
VIRTUALXP-36801, 23:12:39

128 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

121 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

55 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

52 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

53 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

52 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm
...

889 bytes in LogJJ.txt created 23:12:39
-- ok --


Is that better?

Gunther
You have to know the facts before you can distort them.

dedndave

#6
May 29, 2013, 07:46:14 AM
Code Select
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Dave\Desktop\GetFilesEx => getfilesex
Intel(R) Pentium(R) 4 CPU 3.00GHz (MMX, SSE, SSE2, SSE3)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
K7NL, 14:45:42

258 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

253 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

121 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

123 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

119 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

122 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm
...

882 bytes in LogJJ.txt created 2:45:43 PM

Magnum

#7
May 29, 2013, 08:34:41 AM
Code Select

Intel(R) Core(TM)2 Duo CPU     P8600  @ 2.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
21-7, 17:33:57

218 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013 5:08:33 PM \Masm32\Examples\timer_demos
5/28/2013 5:08:33 PM \Masm32\Examples\Bill_Cravener

124 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013 5:08:33 PM \Masm32\Examples\timer_demos
5/28/2013 5:08:33 PM \Masm32\Examples\Bill_Cravener

58 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013 5:08:33 PM \Masm32\Examples\timer_demos
5/28/2013 5:08:33 PM \Masm32\Examples\Bill_Cravener

59 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013 5:08:33 PM \Masm32\Examples\timer_demos
5/28/2013 5:08:33 PM \Masm32\Examples\Bill_Cravener

58 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013 5:08:33 PM \Masm32\Examples\timer_demos
5/28/2013 5:08:33 PM \Masm32\Examples\Bill_Cravener

59 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013 5:08:33 PM \Masm32\Examples\timer_demos
5/28/2013 5:08:33 PM \Masm32\Examples\Bill_Cravener
...

1876 bytes in LogJJ.txt created 5:33:57 PM
-- ok --
Take care,
                   Andy

Ubuntu-mate-18.04-desktop-amd64

http://www.goodnewsnetwork.org

sinsi

#8
May 29, 2013, 01:55:57 PM

Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX)
Windows 8 Pro with Media Center
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
Line 12: Get/SetRegVal failed
, 13:23:33

59 ms for 1000 loops, BasicInfo=1, LargeFetch=2
19/12/2012      4:48:12 PM      \Masm32\Examples\timer_demos
19/12/2012      4:48:12 PM      \Masm32\Examples\poasm

45 ms for 1000 loops, BasicInfo=0, LargeFetch=0
19/12/2012      4:48:12 PM      \Masm32\Examples\timer_demos
19/12/2012      4:48:12 PM      \Masm32\Examples\poasm

51 ms for 1000 loops, BasicInfo=1, LargeFetch=2
19/12/2012      4:48:12 PM      \Masm32\Examples\timer_demos
19/12/2012      4:48:12 PM      \Masm32\Examples\poasm

37 ms for 1000 loops, BasicInfo=0, LargeFetch=0
19/12/2012      4:48:12 PM      \Masm32\Examples\timer_demos
19/12/2012      4:48:12 PM      \Masm32\Examples\poasm

36 ms for 1000 loops, BasicInfo=1, LargeFetch=2
19/12/2012      4:48:12 PM      \Masm32\Examples\timer_demos
19/12/2012      4:48:12 PM      \Masm32\Examples\poasm

34 ms for 1000 loops, BasicInfo=0, LargeFetch=0
19/12/2012      4:48:12 PM      \Masm32\Examples\timer_demos
19/12/2012      4:48:12 PM      \Masm32\Examples\poasm

Siekmanski

#9
May 29, 2013, 04:30:13 PM
Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz (MMX, SSE, SSE2, SSE3, SSSE3)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
SIEKMANSKIPC, 08:25:58

147 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007      10:47:34        \Masm32\Examples\bcraven
24-12-2007      10:47:24        \Masm32\Examples\exampl11

146 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007      10:47:34        \Masm32\Examples\bcraven
24-12-2007      10:47:24        \Masm32\Examples\exampl11

74 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007      10:47:34        \Masm32\Examples\bcraven
24-12-2007      10:47:24        \Masm32\Examples\exampl11

74 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007      10:47:34        \Masm32\Examples\bcraven
24-12-2007      10:47:24        \Masm32\Examples\exampl11

75 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007      10:47:34        \Masm32\Examples\bcraven
24-12-2007      10:47:24        \Masm32\Examples\exampl11

74 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007      10:47:34        \Masm32\Examples\bcraven
24-12-2007      10:47:24        \Masm32\Examples\exampl11
...

880 bytes in LogJJ.txt created 8:25:59
-- ok --
Creative coders use backward thinking techniques as a strategy.

jj2007

#10
May 29, 2013, 07:30:39 PM
I've done some more testing - see new attachment on top of thread, now allowing to drag a folder over the exe to get any folder tested.

The pattern that emerges is that for small folders FindFirstFileEx performs better without the two flags, but for large folders and network drives it can be some 20% or so faster with the flags set. Not on XP, though, this is Win7 only.

AMD Athlon(tm) Dual Core Processor 4450B (MMX, SSE, SSE2, SSE3)
Windows 7 Professional
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags

Commandline=[C:\Masm32\RichMasm], testing [C:\Masm32\RichMasm\*]
...
last #folders=18
Total ms by flag setting, ignoring first 2 runs:
25692 ms for flags OFF
19988 ms for flags ON

Commandline=[?], testing [C:\Masm32\Examples\*]
...
last #folders=18
Total ms by flag setting, ignoring first 2 runs:

1068 ms for flags OFF
1476 ms for flags ON

Commandline=["C:\Program Files"], testing [C:\Program Files\*]
...
last #folders=51
Total ms by flag setting, ignoring first 2 runs:
1870 ms for flags OFF
1681 ms for flags ON

sinsi

#11
May 29, 2013, 08:10:58 PM
From a command prompt:
F:\>GetFilesEx.exe
Commandline=[?], testing [m:\Masm32\Examples\*]
However, it works OK double-clicking from Explorer.

Worked:
F:\>GetFilesEx.exe c:\
Commandline=[c:\], testing [c:\\*]

2 ms for 10 loops, BasicInfo=1, LargeFetch=2  29/05/2013 2:08:35 PM ..formation
1 ms for 10 loops, BasicInfo=0, LargeFetch=0  29/05/2013 2:08:35 PM ..formation
2 ms for 10 loops, BasicInfo=1, LargeFetch=2  29/05/2013 2:08:35 PM ..formation
1 ms for 10 loops, BasicInfo=0, LargeFetch=0  29/05/2013 2:08:35 PM ..formation
1 ms for 10 loops, BasicInfo=1, LargeFetch=2  29/05/2013 2:08:35 PM ..formation
1 ms for 10 loops, BasicInfo=0, LargeFetch=0  29/05/2013 2:08:35 PM ..formation
...

4220429 bytes in  created 9:30:00 AM, last #folders=9
Total ms by flag setting, ignoring first 2 runs:
9 ms for flags OFF
7 ms for flags ON

Failed:
F:\>GetFilesEx.exe c:
Commandline=[c:], testing [m:\Masm32\Examples\*]

dedndave

#12
May 29, 2013, 08:18:59 PM
it tries to run on my G: drive, when masm32 is on my C: drive
i have an older version of masm32 on G:

jj2007

#13
May 29, 2013, 08:27:26 PM
Thanks, Sinsi & Dave. I attached a corrected version above.

(I had assumed the zero arg of the commandline was a full path to the exe, but it's just what you type in :()

dedndave

#14
May 29, 2013, 08:40:23 PM
my bad - i did not put a command line arg   :redface:

now, it says no file for C:\masm32\examples, which is incorrect
Print
Go Up Pages1 2
User actions