Adventures in programming: Treeviews, directory scanning, recursion

[sinsi]

First of all, daydreamer, you really need to learn to write coherent English. It's sometimes very difficult to figure out what you're trying to say.

Don't be a dick. You don't use emojis, which makes some of your posts dick-ish

If you re-scan a drive that you have already scanned, do you replace/free the previous "objects" or just add them all again to your list? There's no way that a treeview uses that much memory for 100 items, whereas if you are adding a new object to your list, that's at least MAX-PATH plus.

[NoCforMe]

First of all, daydreamer, you really need to learn to write coherent English. It's sometimes very difficult to figure out what you're trying to say.

Don't be a dick. You don't use emojis, which makes some of your posts dick-ish

Thank you for sharing that with me. (Again, I don't use emojis; read between the lines.)

If you re-scan a drive that you have already scanned, do you replace/free the previous "objects" or just add them all again to your list? There's no way that a treeview uses that much memory for 100 items, whereas if you are adding a new object to your list, that's at least MAX-PATH plus.

Every new scan fills my list from the beginning, so that's not the problem. Besides, I'm only allocating memory (heap) once at the start of the program.

Specifically:

• I allocate a big heap (100,000 entries) at the start
• For each scan, I start at the start of the heap (the pointer I get back from HeapAlloc())

Did you get that much memory usage (you said 350 MB) with only 100 items? Weird.

Judging from what I'm reading in lots of other forums, treeviews are notorious memory hogs.

[sinsi]

How much memory is 100K objects?
A scan of my C: (Windows and programs only, users are on a different drive) gives 527,708 files and 174,068 folders, a bit more than your 100,000 limit 

read between the lines

I'm a native english speaker and sometimes I don't even see your lines

[NoCforMe]

How much memory is 100K objects?

Welll, since FILEINFO is thus:

Code Select Expand

FILEINFO STRUC
  pathName DB MAX_PATH DUP(?)
  rootLoc DD ?
  textPtr DD ?
  parent DD ?
  tvHandle HWND ?
  flag DD ?
FILEINFO ENDS
then it's 26,000,000 bytes.

A scan of my C: (Windows and programs only, users are on a different drive) gives 527,708 files and 174,068 folders, a bit more than your 100,000 limit 

Yes; well, obviously I've got a ways to go until this is "professional grade". Memory re-allocation is next up on my list.

[sinsi]

[pedant]28,000,000[/pedant]

[NoCforMe]

No, you're absolutely right. Why was I thinking MAX_PATH = 240?

Anyhow.

[NoCforMe]

Memory Issues

Now this is starting to bug me.

I changed the program so it now destroys and re-creates the treeview from scratch each and every time a new scan (selecting a new path) is done. So you'd think this would eliminate the control as the source of escalating memory usage. (DestroyWindow() should completely wipe out any memory allocated by the control, right?

Memory usage still increases with each scan. ?????

Again: I am not allocating any memory apart from a large initial allocation at the beginning. So where is this memory leak? It must be Windows, not me, right? But where?

Another weirdness: My heap allocation is 280,000,000 bytes, and yet when I first run the program, Task Manager only shows a usage of 1.6 MB. Now I read something recently that said that what TM reports isn't completely reliable, but this is not even in the ballpark.

Anyone have any clues as to what's going on here?

BTW, it's easy to re-create the control: GetWindowInfo() is your friend. However, there's yet another needed thing missing from the MASM32 include files:

Code Select Expand

WINDOWINFO STRUCT
  cbSize DD ?
  rcWindow RECT <>
  rcClient RECT <>
  dwStyle DD ?
  dwExStyle DD ?
  dwWindowStatus DD ?
  cxWindowBorders DD ?
  cyWindowBorders DD ?
  atomWindowType ATOM ? ;WORD
  wCreatorVersion DW ?
WINDOWINFO ENDS


[fearless]

Using any images? icons? bitmaps?

[NoCforMe]

Only the ones that are used in the treeview's image lists (3 little itty-bitty 16x16 bitmaps). I just made sure that I was removing both image lists when re-creating the control (using TVM_GETIMAGELIST to get them and then destroying them with ImageList_Destroy(); made absolutely no difference.

I'm displaying bitmap images, sure, using GDI+, but the memory leak persists even if I never show a single image. Other than that, no images (other than the one icon used by the program, and there's no way that's gonna be responsible for a multi-megabyte memory leak!).

Check out this memory-leak report, as well as this one. They're all over the place!

Maybe I need to use a good profiler here? Anyone have any suggestions in that regard?

[NoCforMe]

Plan B:

Maybe use a different control. Not a listview! I hate, hate, hate dealing with that hot mess of a control.

Maybe an owner-drawn listbox. The lowly listbox is one of the most primitive Windows controls, has been there since Petzold days, so it's proven and simple. It wouldn't give the nice sexy appearance that the treeview gives us, but I could probably simulate the tree-structure view with indentation (using owner draw).

Ugh. Not ready to go down that road yet, but may have to soon ...

[sinsi]

A listbox can only have 32767 items that's how primitive it is

[NoCforMe]

Then Plan C: write my own custom control. (I've done it before.) Could be fun.

I'm even thinking of maybe trying to implement my own treeview ...

[fearless]

I couldnt download your previous examples as they flagged by AV. If you have some working example code that someone can look at, might be worth posting that first, to see if anything can be spotted, and it might mean you dont have to go down that road of writing your own control.

[NoCforMe]

Well, here's everything. Have at it. I seriously doubt you'll find a memory leak in here. Of course, you may find any number of other problems ...

It does work, doesn't crash, has a hard limit of 100,000 "objects" (folders+files). Play around with it.

[jj2007]

I did the LPSTR_TEXTCALLBACK thing

That should have solved all your memory problems, as now you only have some minimal "infrastructure" that gets used over and over. The question is what went wrong.