Adventures in programming: Treeviews, directory scanning, recursion

[NoCforMe]

Yes, it would be nice to see at a glance what kind of "object" that is. I'm curious about attaching li'l images anyhow.

I must say that I'm still liking this project. So far all the problems have been self-inflicted, and when they're fixed there's no real mystery about how things work. Even if nothing more comes of this, it's a good learning experience.

[NoCforMe]

OK, last post for tonight. You say you wanted images? You got images. I think it makes it a lot easier to use. LMK what you think.

(The "picture" bitmap is a bit ugly, best I could come up with on short notice. They're all 16x16 bitmaps. Oh, and be sure to scan a directory that has at least 1 empty folder.)

[jj2007]

Note: Long post

Yes indeed

But very helpful. PixOrg.zip (reply #7) works like a charm, compliments

[sinsi]

 

[NoCforMe]

Moving on to other issues:

Stack usage:
Since this program uses recursion (omigod!), stack size becomes an issue, although after thinking about it, maybe not that much of an issue: after all, the depth of recursion is limited in my case to the depth of folder nesting in the file system. Let's say that it can get up to 12 or 15 deep, or maybe even 20: that doesn't seem like a huge amount of stack space is needed. (On the other hand, I did get a couple failures when testing due to stack overflow.)

So I'm thinking about doing some testing to try to determine how much stack space is actually needed. One idea is to mark the stack at program start, then scan the stack after running to see how much actually got used. Maybe something like this to mark it:

Code Select Expand

MOV EBX, ESP ;save it
MOV EDX, ESP
MOV ECX, <stack size - a little something>
sloop: MOV DWORD PTR [EDX]. 0BADF00D ;"PUSH"
SUB EDX, 4
LOOP sloop
MOV ESP, EBX ;restore it
then later I can scan the stack looking for the first marked value (0BADF00D) which will give me at least an approximate number for usage.

This won't mess up my program, will it? (Just for research, won't be in the "release" version if that ever happens.)

[NoCforMe]

Hmm; I tried it. No harm to the program. Looks like stack usage was right around 4,000 DWORDs, so not far from what I had guessed (I set .stack to 20K, 20 * 1,024).

I actually got a stack overflow when it was set to half of that.

Stack checking code:

Code Select Expand

mov edx, esp
sub edx, 256
mov ecx, ($stackSize / 4) - 256
xor eax, eax
cloop: cmp dword ptr [edx], 0BADF00Dh
je xxx
inc eax
sub edx, 4
loop cloop
xxx: invoke wsprintf, addr buffer, offset stackfmt, eax
invoke MessageBox, NULL, addr buffer, NULL, MB_OK


[NoCforMe]

New version attached. Bugs fixed, made more bulletproof. I think. I hope.

I made one change that should help the "takes a long time to grind through a disk" problem. I won't tell you what it is, so you'll need to at least be curious enough to download it and try it. (Right, JJ?) Comments welcome as always. (I didn't use Greenhorn's suggestion of doing incremental scans; although sounds like it has merit, it would be complicated to implement. Maybe later?)

Development is at a standstill at this point, since I'm not sure where I'm going next. I need to talk to the "customer" (a friend who casually mentioned they might like such a program) to find out what they really want. No problem, as I've been having fun with this project.

[jj2007]

I made one change that should help the "takes a long time to grind through a disk" problem.

The MsgBox "Maximum # of folders + files (100,000) hit."?

[NoCforMe]

Uh, yeah, another Issue.

I've got to allocate some finite amount of memory to start, right? Chose enough for 100,000 "objects" (files/folders). Now, in a Real Product, one wouldn't just throw one's hands up when that limit is hit: one would allocate some more memory and continue on.

Coming in version 2 ... hey, at least it didn't crash on you!

[sinsi]

Do you ever release memory? The usage only increases, even if I selected a drive and then reselected it.
It got to about 350MB before I tried to scan C: and got the limit message.

[NoCforMe]

I'm pretty sure that memory is being used by the treeview, since I only do one allocation for my file list heap at the start. Not sure how to release treeview memory: destroy and re-create the control each time? (I'm using TVM_DELETEITEM to clear the control at each new search, but there are other ways to do that.)

Just found this article (a Microsoft knowledge base posting) which says that you can reduce treeview memory usage by using the LPSTR_TEXTCALLBACK value instead of explicitly passing a string for text display; that way the caller is responsible for the string storage space, not the control. Since each treeview item has a pointer to my file list, that should be easy to implement.

Not sure if that would help the cumulative memory usage problem, though.

[NoCforMe]

Well, I did the LPSTR_TEXTCALLBACK thing (easy, just had to add a handler and fill in the NMTVDISPINFO structure). It made a tiny improvement in memory usage. But it didn't solve the problem of increasing memory usage every time you do a new folder scan. I'm thinking that's gonna require destroying and re-creating the control every time. Ugh.

Drive scanned	Non-callback	Callback
C:	115,260	112,136
D:	216,852	214,828

[NoCforMe]

Maybe the answer is simple. Found this in an online forum:

Displaying half a million records in a treeview is not a great idea.

But it's such a nice control. I really like it. Hate to have to give it up.

[daydreamer]

I'm pretty sure that memory is being used by the treeview, since I only do one allocation for my file list heap at the start. Not sure how to release treeview memory: destroy and re-create the control each time? (I'm using TVM_DELETEITEM to clear the control at each new search, but there are other ways to do that.)

Just found this article (a Microsoft knowledge base posting) which says that you can reduce treeview memory usage by using the LPSTR_TEXTCALLBACK value instead of explicitly passing a string for text display; that way the caller is responsible for the string storage space, not the control. Since each treeview item has a pointer to my file list, that should be easy to implement.

Not sure if that would help the cumulative memory usage problem, though.

Doesn't it suggest there is a bug like not reset pointer to start of allocated memory or something ?

[NoCforMe]

Doesn't it suggest there is a bug like not reset pointer to start of allocated memory or something ?

First of all, daydreamer, you really need to learn to write coherent English. It's sometimes very difficult to figure out what you're trying to say.

Resetting pointer? Bug? But who?--me, or Windows? I'm not doing anything with memory (I make one big allocation at the start for my file list, and that's it), so it's not my fault. So far as Windows goes, the treeview seems like a pretty robust and well-tested control, except that it's a real memory hog. (This seems confirmed by the number of posts you find online about memory problems with it.) If only there were some way to "reset pointer to start of allocated memory", or something. But there are apparently no such hooks into the code.

I'm really thinking destroying/re-creating is the only way to go. It's not that hard; I just have been reluctant to code it. (It's not that fun.)