Author Topic: FindFirstFileEx, FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH (Read 17141 times)

jj2007 · « **on:** May 28, 2013, 08:58:06 PM »

I am trying to speed up file searches but no luck so far:

AMD Athlon(tm) Dual Core Processor 4450B (MMX, SSE, SSE2, SSE3) Windows 7 Professional Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags 21369 µs for 100 loops, BasicInfo=1, LargeFetch=1 23.7.12 14:24:04 \Masm32\Examples\exampl01 20.2.12 18:15:47 \Masm32\Examples\enumerate 19708 µs for 100 loops, BasicInfo=0, LargeFetch=0 23.7.12 14:24:04 \Masm32\Examples\exampl01 20.2.12 18:15:47 \Masm32\Examples\enumerate 19834 µs for 100 loops, BasicInfo=1, LargeFetch=1 23.7.12 14:24:04 \Masm32\Examples\exampl01 20.2.12 18:15:47 \Masm32\Examples\enumerate 20297 µs for 100 loops, BasicInfo=0, LargeFetch=0 23.7.12 14:24:04 \Masm32\Examples\exampl01 20.2.12 18:15:47 \Masm32\Examples\enumerate 16150 µs for 100 loops, BasicInfo=1, LargeFetch=1 23.7.12 14:24:04 \Masm32\Examples\exampl01 20.2.12 18:15:47 \Masm32\Examples\enumerate 18467 µs for 100 loops, BasicInfo=0, LargeFetch=0 23.7.12 14:24:04 \Masm32\Examples\exampl01 20.2.12 18:15:47 \Masm32\Examples\enumerate

I've seen claims for miraculous improvements but my OS fails to honour these flags :(
Grateful for tests and ideas.

P.S.: I am aware of Lingo's stuff but it crashes. Probably too many GetProcAddress with ordinals ;)

dedndave · « **Reply #1 on:** May 28, 2013, 11:05:14 PM »

Code: [Select]

Intel(R) Pentium(R) 4 CPU 3.00GHz (MMX, SSE, SSE2, SSE3)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags

190 µs for 100 loops, BasicInfo=1, LargeFetch=1

3100 µs for 100 loops, BasicInfo=0, LargeFetch=0

251 µs for 100 loops, BasicInfo=1, LargeFetch=1

3045 µs for 100 loops, BasicInfo=0, LargeFetch=0

254 µs for 100 loops, BasicInfo=1, LargeFetch=1

3052 µs for 100 loops, BasicInfo=0, LargeFetch=0

jj2007 · « **Reply #2 on:** May 29, 2013, 12:59:57 AM »

Thanks, Dave.

This is really messy (MSDN FindFirstFileEx):

Quote

FIND_FIRST_EX_LARGE_FETCH .. and Windows XP: This value is not supported until Windows Server 2008 R2 and Windows 7."

~~You just demonstrated the contrary~~
EDIT: No, that was just a bug in the first version.

Now why is it slow on my Win7-32 and -64 systems but fast on XP? ::)

To get the computer name, I added
TryRTE RegX2, con
Let edi=GetRegVal("HKLM\SYSTEM\CurrentControlSet\Control\ComputerName\ComputerName", "ComputerName")
RegX2:
Works fine on W7-32 but requires admin privileges on W7-64; however, when I run it as admin, it does no longer read the network drive... oh my dear friends in Redmond... :icon_mrgreen:

Gunther · « **Reply #3 on:** May 29, 2013, 03:58:44 AM »

Jochen,

the results:

Code: [Select]

Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SS
E4.2, AVX)
Windows 7 Professional
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags

1112 µs for 100 loops, BasicInfo=1, LargeFetch=1
01.01.1601      02:00:00
01.01.1601      02:00:00

804 µs for 100 loops, BasicInfo=0, LargeFetch=0
01.01.1601      02:00:00
01.01.1601      02:00:00

1042 µs for 100 loops, BasicInfo=1, LargeFetch=1
01.01.1601      02:00:00
01.01.1601      02:00:00

753 µs for 100 loops, BasicInfo=0, LargeFetch=0
01.01.1601      02:00:00
01.01.1601      02:00:00

1002 µs for 100 loops, BasicInfo=1, LargeFetch=1
01.01.1601      02:00:00
01.01.1601      02:00:00

919 µs for 100 loops, BasicInfo=0, LargeFetch=0
01.01.1601      02:00:00
01.01.1601      02:00:00
-- ok --

Gunther

jj2007 · « **Reply #4 on:** May 29, 2013, 05:46:50 AM »

Thanxalot, Dave and Gunther. In the meantime, I fixed the bug that made it so fast on XP :icon_mrgreen:
; perform a dummy search to find out if the OS supports the fast flags: FindExInfoBasic=1 invoke FindFirstFileEx, chr$("C:\NoFFx.Yet"), FindExInfoBasic, offset wfd, 0, 0, FIND_FIRST_EX_LARGE_FETCH invoke GetLastError <<<<<<<< I had forgotten this one :redface: .if eax!=ERROR_INVALID_PARAMETER add dword ptr [esi], FindExInfoBasic ; ffxInfoLevel add dword ptr [esi+4], FIND_FIRST_EX_LARGE_FETCH ; ffxAdditionalFlags .endif

New version attached at top of thread.

@Gunther: It searches \Masm32\Examples\* - since you had no results, that was probably the wrong drive or puter. The new version complains with a MsgBox if it doesn't find anything.

XP results:
Intel(R) Celeron(R) M CPU 420 @ 1.60GHz (MMX, SSE, SSE2, SSE3)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
MSI, 21:47:41
253 ms for 1000 loops, BasicInfo=0, LargeFetch=0 02.03.2013 00:51:53 \Masm32\Examples\exampl01 03.11.2012 12:16:41 \Masm32\Examples\Bill_Cravener 288 ms for 1000 loops, BasicInfo=0, LargeFetch=0 02.03.2013 00:51:53 \Masm32\Examples\exampl01 03.11.2012 12:16:41 \Masm32\Examples\Bill_Cravener 130 ms for 1000 loops, BasicInfo=0, LargeFetch=0 02.03.2013 00:51:53 \Masm32\Examples\exampl01 03.11.2012 12:16:41 \Masm32\Examples\Bill_Cravener 127 ms for 1000 loops, BasicInfo=0, LargeFetch=0 02.03.2013 00:51:53 \Masm32\Examples\exampl01 03.11.2012 12:16:41 \Masm32\Examples\Bill_Cravener

Flags are never set, i.e. it fails on XP as documented.

Gunther · « **Reply #5 on:** May 29, 2013, 07:14:48 AM »

Jochen,

here are the results from my virtual machine:

Code: [Select]

Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SS
E4.2, AVX)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
VIRTUALXP-36801, 23:12:39

128 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

121 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

55 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

52 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

53 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm

52 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12.01.2013      22:03:43        \Masm32\Examples\timer_demos
12.01.2013      22:03:43        \Masm32\Examples\poasm
...

889 bytes in LogJJ.txt created 23:12:39
-- ok --

Is that better?

Gunther

dedndave · « **Reply #6 on:** May 29, 2013, 07:46:14 AM »

Code: [Select]

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Dave\Desktop\GetFilesEx => getfilesex
Intel(R) Pentium(R) 4 CPU 3.00GHz (MMX, SSE, SSE2, SSE3)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
K7NL, 14:45:42

258 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

253 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

121 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

123 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

119 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm

122 ms for 1000 loops, BasicInfo=0, LargeFetch=0
12/19/2012      8:25:23 AM      \Masm32\Examples\static
12/10/2012      10:30:42 AM     \Masm32\Examples\poasm
...

882 bytes in LogJJ.txt created 2:45:43 PM

Magnum · « **Reply #7 on:** May 29, 2013, 08:34:41 AM »

Code: [Select]

Intel(R) Core(TM)2 Duo CPU     P8600  @ 2.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
21-7, 17:33:57

218 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013	5:08:33 PM	\Masm32\Examples\timer_demos
5/28/2013	5:08:33 PM	\Masm32\Examples\Bill_Cravener

124 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013	5:08:33 PM	\Masm32\Examples\timer_demos
5/28/2013	5:08:33 PM	\Masm32\Examples\Bill_Cravener

58 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013	5:08:33 PM	\Masm32\Examples\timer_demos
5/28/2013	5:08:33 PM	\Masm32\Examples\Bill_Cravener

59 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013	5:08:33 PM	\Masm32\Examples\timer_demos
5/28/2013	5:08:33 PM	\Masm32\Examples\Bill_Cravener

58 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013	5:08:33 PM	\Masm32\Examples\timer_demos
5/28/2013	5:08:33 PM	\Masm32\Examples\Bill_Cravener

59 ms for 1000 loops, BasicInfo=0, LargeFetch=0
5/28/2013	5:08:33 PM	\Masm32\Examples\timer_demos
5/28/2013	5:08:33 PM	\Masm32\Examples\Bill_Cravener
...

1876 bytes in LogJJ.txt created 5:33:57 PM
-- ok --

sinsi · « **Reply #8 on:** May 29, 2013, 01:55:57 PM »

Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX) Windows 8 Pro with Media Center Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags Line 12: Get/SetRegVal failed , 13:23:33 59 ms for 1000 loops, BasicInfo=1, LargeFetch=2 19/12/2012 4:48:12 PM \Masm32\Examples\timer_demos 19/12/2012 4:48:12 PM \Masm32\Examples\poasm 45 ms for 1000 loops, BasicInfo=0, LargeFetch=0 19/12/2012 4:48:12 PM \Masm32\Examples\timer_demos 19/12/2012 4:48:12 PM \Masm32\Examples\poasm 51 ms for 1000 loops, BasicInfo=1, LargeFetch=2 19/12/2012 4:48:12 PM \Masm32\Examples\timer_demos 19/12/2012 4:48:12 PM \Masm32\Examples\poasm 37 ms for 1000 loops, BasicInfo=0, LargeFetch=0 19/12/2012 4:48:12 PM \Masm32\Examples\timer_demos 19/12/2012 4:48:12 PM \Masm32\Examples\poasm 36 ms for 1000 loops, BasicInfo=1, LargeFetch=2 19/12/2012 4:48:12 PM \Masm32\Examples\timer_demos 19/12/2012 4:48:12 PM \Masm32\Examples\poasm 34 ms for 1000 loops, BasicInfo=0, LargeFetch=0 19/12/2012 4:48:12 PM \Masm32\Examples\timer_demos 19/12/2012 4:48:12 PM \Masm32\Examples\poasm

Siekmanski · « **Reply #9 on:** May 29, 2013, 04:30:13 PM »

Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (MMX, SSE, SSE2, SSE3, SSSE3)
Microsoft Windows XP
Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags
SIEKMANSKIPC, 08:25:58

147 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007 10:47:34 \Masm32\Examples\bcraven
24-12-2007 10:47:24 \Masm32\Examples\exampl11

146 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007 10:47:34 \Masm32\Examples\bcraven
24-12-2007 10:47:24 \Masm32\Examples\exampl11

74 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007 10:47:34 \Masm32\Examples\bcraven
24-12-2007 10:47:24 \Masm32\Examples\exampl11

74 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007 10:47:34 \Masm32\Examples\bcraven
24-12-2007 10:47:24 \Masm32\Examples\exampl11

75 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007 10:47:34 \Masm32\Examples\bcraven
24-12-2007 10:47:24 \Masm32\Examples\exampl11

74 ms for 1000 loops, BasicInfo=0, LargeFetch=0
24-12-2007 10:47:34 \Masm32\Examples\bcraven
24-12-2007 10:47:24 \Masm32\Examples\exampl11
...

880 bytes in LogJJ.txt created 8:25:59
-- ok --

jj2007 · « **Reply #10 on:** May 29, 2013, 07:30:39 PM »

I've done some more testing - see new attachment on top of thread, now allowing to drag a folder over the exe to get any folder tested.

The pattern that emerges is that for small folders FindFirstFileEx performs better without the two flags, but for large folders and network drives it can be some 20% or so faster with the flags set. Not on XP, though, this is Win7 only.

AMD Athlon(tm) Dual Core Processor 4450B (MMX, SSE, SSE2, SSE3) Windows 7 Professional Test FindExInfoBasic and FIND_FIRST_EX_LARGE_FETCH flags Commandline=[C:\Masm32\RichMasm], testing [C:\Masm32\RichMasm\*] ... last #folders=18 Total ms by flag setting, ignoring first 2 runs: 25692 ms for flags OFF 19988 ms for flags ON Commandline=[?], testing [C:\Masm32\Examples\*] ... last #folders=18 Total ms by flag setting, ignoring first 2 runs: 1068 ms for flags OFF 1476 ms for flags ON Commandline=["C:\Program Files"], testing [C:\Program Files\*] ... last #folders=51 Total ms by flag setting, ignoring first 2 runs: 1870 ms for flags OFF 1681 ms for flags ON

sinsi · « **Reply #11 on:** May 29, 2013, 08:10:58 PM »

From a command prompt:
F:\>GetFilesEx.exe Commandline=[?], testing [m:\Masm32\Examples\*]
However, it works OK double-clicking from Explorer.

Worked:
F:\>GetFilesEx.exe c:\ Commandline=[c:\], testing [c:\\*] 2 ms for 10 loops, BasicInfo=1, LargeFetch=2 29/05/2013 2:08:35 PM ..formation 1 ms for 10 loops, BasicInfo=0, LargeFetch=0 29/05/2013 2:08:35 PM ..formation 2 ms for 10 loops, BasicInfo=1, LargeFetch=2 29/05/2013 2:08:35 PM ..formation 1 ms for 10 loops, BasicInfo=0, LargeFetch=0 29/05/2013 2:08:35 PM ..formation 1 ms for 10 loops, BasicInfo=1, LargeFetch=2 29/05/2013 2:08:35 PM ..formation 1 ms for 10 loops, BasicInfo=0, LargeFetch=0 29/05/2013 2:08:35 PM ..formation ... 4220429 bytes in created 9:30:00 AM, last #folders=9 Total ms by flag setting, ignoring first 2 runs: 9 ms for flags OFF 7 ms for flags ON

Failed:
F:\>GetFilesEx.exe c: Commandline=[c:], testing [m:\Masm32\Examples\*]

dedndave · « **Reply #12 on:** May 29, 2013, 08:18:59 PM »

it tries to run on my G: drive, when masm32 is on my C: drive
i have an older version of masm32 on G:

jj2007 · « **Reply #13 on:** May 29, 2013, 08:27:26 PM »

Thanks, Sinsi & Dave. I attached a corrected version above.

(I had assumed the zero arg of the commandline was a full path to the exe, but it's just what you type in :()

dedndave · « **Reply #14 on:** May 29, 2013, 08:40:23 PM »

my bad - i did not put a command line arg :redface:

now, it says no file for C:\masm32\examples, which is incorrect

The MASM Forum

News: