I just made a big mistake today.
I had created a huge directory with 869 GB in 131072 files. I had done this on my 4 TB backup drive (4KB cluster size) due to the large size of the data. The files were potentially fragmented and I wanted to defrag them. I tried my JKDefrag (created under Win XP, but this time on a my Win7 laptop), but it didn't work - didn't crash or anything else, just didn't defrag the drive, didn't display the progress, and didn't display the first 25 largest files and their sizes. I didn't want to defrag the entire drive using the Windows tools (could see no way to just defrag the directory as I could under XP JKDefrag). So I decided to copy all of the files from one of my 4 TB drives to the other 4 TB drive. Explorer will create exact sized unfragmented file under file copy to a different drive if there are enough available clusters.
Now my laptop has 4 USB connections, 2 are USB3 and 2 are USB2. I use the USB2 connections for my mouse and my Verizon Modem, and have my 400 GB (USB3) program drive connected to the USB3 port. I wanted to copy from a 4 TB drive to another 4 TB drive, so I plugged one in the open USB3 port, and just dismounted the Modem and plugged in the 4 TB drive. I started the file copy (actually copied the directory from one drive to the other), and did not pay much attention to the process, and went about doing other things while the files copied.
About 4 hours later I went to look at the progress.
Big mistake! I had been copying for 4 hours and still had 7 hours to go! At first I could not understand what was happening because I had done this before and the time was just about 5.5 hours. Then it dawned on me, I was being throttled by the USB2 connection (in the prior attempt I had used both of the USB3 ports). I did not want to stop the copy and then retry it after swapping the USB2 connection for the USB3 connection (by dismounting my 400 GB programing drive). To be clean, and defragemented, you have to also delete the files that were copied, and this can take hours in and of itself, so I just let the copy continue. It should be complete by midnight, much past my 10 PM normal bedtime ( I do not leave the system up 24-7 - it is in the bedroom and I don't like the glowing lights).
Moral of the story: be sure about your connections before you start a huge (869 GB) directory copy, and make sure you start it early. This goes along with the old adage "When presented with the necessity of swallowing two live frogs before breakfast, NEVER make the mistake of swallowing the smallest one first!"
Dave.
being as the drives are > ~2 tb, they are GUID Partition Table (GPT) formatted, rather than MBR
XP doesn't support GPT directly, although there are drivers available
but, that may explain the defrag issue
i will give you a heads-up....
don't connect a GPT drive to an XP system without GPT support
it is possible that you lose a bunch of data
and, you may have trouble getting it to be a 4 tb drive, again :redface:
You may have still had speed issues going USB3->USB3, if they are on the same controller I think the bandwidth is shared. Still, half of 100MB/s is better than 20MB/s.
>you have to also delete the files that were copied
If that is all there is on the drive, a quick format is much...erm...quicker.
Sinsi,
Unfortunately, there were other backups on it.
Dave,
Thanks for the info, will keep it off of the XP system.
Dave.
I am wondering if having a couple of separate 500 Gb or 1Tb drives would be faster for what you are wanting ?
I don't know how many physical drives XP can support.
I stopped using MyDefrag because you have to have a minimum, as yet unknown, amount of free space to move very large files around.
MyDefrag could not determine that and it got to 98% defragged and would not go higher.
Cancellation that operation was very costly.
Good luck,
Andy
Andy,
I have copied both sets of 131072 files to the alternate drives, and deleted both sets of 256 files. Since there is so much space available on the drives, I get unfragmented files.
I deleted the 2 sets of fragmented files, and I am now in the process of copying the unfragmented files back to the alternate drives (as backups).
I noticed one thing about the speed of copying. One set of 131072 files were all the same size, and they copied at a speed of ~ 35 MBS. The other set of 131072 files are of varying sizes (from 16 KB to 6000 kb), and the copy speed is about 16 MBS (at least when copying the shortest files at the beginning), but never getting much faster than about 18 MBS ( I did not pay rapt attention to the whole process - over 6 hours).
Now, the two USP3 ports are probably on an internal hub and can only transfer data one way at a time (can't have an input and output going on at the same time). It almost appears that the copy process sets up internal buffering based on the initial file sizes, and even when larger files are encountered, it still seems to be throttled down to the slower speed for the smaller files (as if the number of buffers was selected for the smaller sized file, never to be increased). There should be plenty of memory available, 6 GB, and I am not doing anything else on the system while the copies are going on.
Another note for Dave:
I checked the Seagate 4 TB drives and they specify that the drives are good for 8,7 V, and XP. It may be that only later XP SPs support GPT formatted drives. I will check with Seagate. My old XP system is forever stuck on SP1. I tried to install SP2 (or was it SAP2 and SP3 - matters not). I tried the SP install and it crashed, and trying to backout to the most recent checkpoint also didn't work. Luckly, I had taken a Disk Image backup just before trying the SP install, so I backed it out and forgot about it. I had been trying to install Office 10.0 and it required the later SP, so the other half is stuck with Office 2003.
Dave.
Quote from: KeepingRealBusy on September 20, 2013, 08:25:51 AMso the other half is stuck with Office 2003.
Which is better than more recent versions IMHO ;-)
My office nerds tell me that Linux is a lot faster in copying files than Windows Explorer. Have you thought of writing your own file copy routine? Fat buffer for reading in two gigs of files etc? With so many files, it might be worth a try.
http://superuser.com/questions/213954/why-is-windows-explorer-a-lot-slower-than-filezilla-when-doing-ftp-transfers
How about good old Command-Line Xcopy? (http://stackoverflow.com/questions/1329/what-is-a-better-file-copy-alternative-than-the-windows-default) With S: being the source and T: the target:
xcopy /K /R /E /I /S /C /H /G /X /Y s:\*.* t:\
Or, even better: http://www.raymond.cc/blog/12-file-copy-software-tested-for-fastest-transfer-speed/2/
Scroll down to see "The Results and Findings".
It seems FastCopy (http://ipmsg.org/tools/fastcopy.html.en) would be good for your case.
I must be doing something wrong, I regularly use WINFILE from NT4 to copy very large amounts of data from one disk to another and it rips through multiple gigabytes of data at a pleasantly fast rate. It gets a lot slower when it is done across a network even though I have a reliable gigabit network across my computers. Some years ago I had 2 out of 3 machines starting to fail and one dropped its network speed from a gigabit to 100 mbit and it took days to back up all of the data to another machine. If you can manage it, take the disk out of the box to back up and attach it to the machine where you want the data and you will get the data transferred at SATA rate rather than network speed or the even slower USB2. You will do better with USB3 if the hardware on both ends support it.
I have a gadget that will plug into either a SATA2 or a USB2 which you place the disk into the top and it is a reasonable compromise. For the SATA you need an external SATA cable but they were easy enough to get.
Quote from: hutch-- on September 20, 2013, 11:04:23 AM
I must be doing something wrong, I regularly use WINFILE
Hutch,
I was a real fan of WinFile, then 2x (very similar) took over, but both have a problem with slow network drives, so nowadays I believe in FreeCommander ;-)
Having read some pages of comments in various forums over the last hour, it seems that FastCopy is the best choice for copying large amounts of files, with TeraCopy following as #2.
Quote from: KeepingRealBusy on September 20, 2013, 08:25:51 AM
I noticed one thing about the speed of copying. One set of 131072 files were all the same size, and they copied at a speed of ~ 35 MBS. The other set of 131072 files are of varying sizes (from 16 KB to 6000 kb), and the copy speed is about 16 MBS (at least when copying the shortest files at the beginning), but never getting much faster than about 18 MBS ( I did not pay rapt attention to the whole process - over 6 hours).
Now, the two USP3 ports are probably on an internal hub and can only transfer data one way at a time (can't have an input and output going on at the same time). It almost appears that the copy process sets up internal buffering based on the initial file sizes, and even when larger files are encountered, it still seems to be throttled down to the slower speed for the smaller files (as if the number of buffers was selected for the smaller sized file, never to be increased). There should be plenty of memory available, 6 GB, and I am not doing anything else on the system while the copies are going on.
Dave, probably the slowdown while copy of the small files goes caused not by hardware but software reason: the OS takes most of the time to read file table, locate the file, and make changes in the target file system. The copy itself takes too small percentage of time in this process due to small size of a file. So, having copying a bunch of small files you are getting noticeable slowdown.
In days of old when REAL MEN[tm] used to roll their own file copy routines, you would do a test to see how much memory you could safely allocate, allocate just under that then do very long reads and writes to cut down on the shuffle between the two. In modern 32 bit Windows you have the phenomenon of disk caching and lazy writes that make the first few big files fast then it progressively gets slower so if you really need very fast disk copy routines you have to turn off the lazy writes (file flushing), look for the lowest level of disk reads and writes and use very large memory buffers. Win64 appears to be a bit faster than Win32 which makes sense but I confess I have few problems with fast disk copying so I rarely ever bother to look for other techniques.
Windows also copies multiple files at once asynchronously, this can impact times when smaller files are copied.
Quote from: Antariy on September 20, 2013, 12:35:14 PM
Quote from: KeepingRealBusy on September 20, 2013, 08:25:51 AM
I noticed one thing about the speed of copying. One set of 131072 files were all the same size, and they copied at a speed of ~ 35 MBS. The other set of 131072 files are of varying sizes (from 16 KB to 6000 kb), and the copy speed is about 16 MBS (at least when copying the shortest files at the beginning), but never getting much faster than about 18 MBS ( I did not pay rapt attention to the whole process - over 6 hours).
Now, the two USP3 ports are probably on an internal hub and can only transfer data one way at a time (can't have an input and output going on at the same time). It almost appears that the copy process sets up internal buffering based on the initial file sizes, and even when larger files are encountered, it still seems to be throttled down to the slower speed for the smaller files (as if the number of buffers was selected for the smaller sized file, never to be increased). There should be plenty of memory available, 6 GB, and I am not doing anything else on the system while the copies are going on.
Dave, probably the slowdown while copy of the small files goes caused not by hardware but software reason: the OS takes most of the time to read file table, locate the file, and make changes in the target file system. The copy itself takes too small percentage of time in this process due to small size of a file. So, having copying a bunch of small files you are getting noticeable slowdown.
Alex,
I thought that too, until it started copying the larger files (at the end of the file list), but the speed did not increase very much ( from 15.6 KBS to about 18.5 KBS), whereas when copying the 131072 files all the same size (6,152 KB) the copy was running at 56 KBS. I just don't know.
JJ,
I'm not sure XCOPY will copy without fragmenting, i.e. does it create file, then copy a 64 KB block, then copy more 64 KB blocks, etc, each time the file must be grown with a new cluster set, and maybe the current allocation cannot be grown? At least Explorer seems to create the file at the required size, then copy. Only if a single cluster set cannot be allocated because of other fragmented allocations does the file seem to get fragmented and multiple fragments created.
Note, I am doing the copies to force defragmentation since I cannot seem to be able to use JKDefrag and I do not want to spend hours and hours defragging the entire 4 TB disk, and the standard Win7 tools do not have the ability to defrag a single directory, at least not that I can see.
I noted another Explorer error (incidental since they only reflect problems in the elapsed time remaining calculations in the copy message box, but still errors). I took screen shots when I saw this and have attached them below. The copy started out reporting about 3.5 hours to do the copy, then after several hours, I was shocked to see it reporting 14 hours remaining, and I took a screen shot, then shortly thereafter, I got another report that the time was now only about 1 hour. The time of saving the first screen shot was at 8:13 AM (reported 14 hours ), the second was at 8:24 AM (reported 1 hour).
Dave.
Quote from: sinsi on September 20, 2013, 01:29:03 PM
Windows also copies multiple files at once asynchronously, this can impact times when smaller files are copied.
Win7 Explorer does simultaneous copy? Obviously this should make the seeking for the small files to take much time than straight reading the data. Interesting how it will go if the source (especially) and the target (it depends) drives will have large cluster size... probably large source cluster size would have speedup things, especially if the files are fragmented and are not "too small" - i.e. near to the size of the cluster or bigger than it in times. This will not eleminate the seeking on the file table, but will decrease the amount of seeks for the next cluster.
Alex,
Win7 seems to copy the files sequentially, at least the files have increasing file names and they are displayed in order as the copy proceeds (at least some of the names are displayed - happens too rapidly).
Dave.
Quote from: KeepingRealBusy on September 20, 2013, 01:46:17 PM
I thought that too, until it started copying the larger files (at the end of the file list), but the speed did not increase very much ( from 15.6 KBS to about 18.5 KBS), whereas when copying the 131072 files all the same size (6,152 KB) the copy was running at 56 KBS. I just don't know.
Ah... then I don't know the reason, too. Just a thought maybe the system cache "overrun" - it was flooded by previous files, so Windows did not hurried to flush it... BTW, you can check that with the program in my signature - RAM Clear. When you do copy of much files, and get a slowdown somewhere in the middle - look on the indicator of the free memory, if it's low, then it obviously the caching problems of the lazy flusher. Just "for fun" you may try to start an "optimization" of, let's say, half of a memory, and don't interrupt the copy process, all things will seem to "hang" probably, but after optimization the copy should go much faster for the some time - until the system cache will be flooded again. Explorer uses strange "try to buffer in memory the entire things that a lot bigger than memory" strategy ::)
(Just a note: RAM Clear needs to be swithed into Expert mode and then into Advanced algorithm mode, to free more than 1,5-2 GB of RAM)
Quote from: KeepingRealBusy on September 20, 2013, 02:04:26 PM
Win7 seems to copy the files sequentially, at least the files have increasing file names and they are displayed in order as the copy proceeds (at least some of the names are displayed - happens too rapidly).
Then probably the problem is in too greedy and lazy bufferization that Explorer implements.
What if you zipped all those files and sent one big one and then decompressed it ?
I am going to test that out myself to see because I am curious.
Andy
Linux is an option too, if you have it.
I found a lot of things are faster using it.
Quote from: Magnum on September 21, 2013, 12:10:49 AM
What if you zipped all those files and sent one big one and then decompressed it ?
I am going to test that out myself to see because I am curious.
Andy
Linux is an option too, if you have it.
I found a lot of things are faster using it.
Andy,
Thank you for the suggestions.
Two or more things come to mind. There would be a huge time involved in both zipping the files, and then later in unzipping them, remember we are talking about 500 + GB of data. I do not know if winzip even supports such a large zip, and also I do not know what Win7 uses in its "send to compressed folder" function and whether it would support such large inputs. Then finally, how does the unzip function really work? Will it create files with the known file size and then copy the content, or will it create the file and then start copying one huge block after another until the file is all copied. This can lead to fragmentation.
I do not have Linux, and do not plan to learn to use it.
Dave.
Quote from: jj2007 on September 20, 2013, 10:08:11 AM
Quote from: KeepingRealBusy on September 20, 2013, 08:25:51 AMso the other half is stuck with Office 2003.
Which is better than more recent versions IMHO ;-)
My office nerds tell me that Linux is a lot faster in copying files than Windows Explorer. Have you thought of writing your own file copy routine? Fat buffer for reading in two gigs of files etc? With so many files, it might be worth a try.
http://superuser.com/questions/213954/why-is-windows-explorer-a-lot-slower-than-filezilla-when-doing-ftp-transfers
How about good old Command-Line Xcopy? (http://stackoverflow.com/questions/1329/what-is-a-better-file-copy-alternative-than-the-windows-default) With S: being the source and T: the target:
xcopy /K /R /E /I /S /C /H /G /X /Y s:\*.* t:\
Or, even better: http://www.raymond.cc/blog/12-file-copy-software-tested-for-fastest-transfer-speed/2/
Scroll down to see "The Results and Findings". It seems FastCopy (http://ipmsg.org/tools/fastcopy.html.en) would be good for your case.
JJ,
I downloaded FastCopy and started examining the source code, and several things popped out. They create a file structure for all files all at once - I do not know if windows can support 131072 open files all at the same time. Then it tries to copy the files all at the same time with multiple threads. The copies are made (for any single file) with multiple block transfers, growing each file with each block. With multiple files being copied at the same time, this can only cause massive fragmentation.
No thank you.
Dave.
Dave,
I didn't dig into the source code, sorry. However, the forums I studied put FastCopy on rank 1, followed by TeraCopy (the most popular tool). Among the strong points, low resource use was frequently mentioned.
Wilders: (http://www.wilderssecurity.com/showthread.php?t=231048)
Quote
The fastest for me is FastCopy and also, important, it doesn't make fragmented files like TeraCopy v2 and BurstCopy (tested with Defraggler)
JJ,
Maybe I have to study the code some more, to see if they somehow can create a full sized file before copying the content. Maybe by creating the file, then seeking to the last BYTE, and writing a 0, then seeking to the front and starting the copy. Hard to follow the CPP code, especially with the unreadable "commentary".
Dave.
Dave,
What is the MTBF for your drives ?
My thoughts on fragmentation.
I used to defrag once a week.
I went to once a month and did not notice any decrease in speed of my system.
Dave, you might not want to read below the line.
-------------------------------------------------------------------
For those who have been, or may be considering Linux.
Quote:
Originally Posted by yancek View Post
"they don't get fragmented until a drive is almost full."
False.
They get fragmented (though admittedly not *that much* as in windows). They key point are I/O schedulers, that reorder operations in a smart manner. So the whole point is not the lack of fragmentation, but the smart scheduling of the I/O which prevents all the stress associated typically to fragmented devices making the performance penalty pointless. However this can vary from fs to fs. Reiserfs has known problems with fragmentation, but the rest of the fs's should be fine unless the disk is almost full as you say. Even fat will perform ok on linux, unlike windows xp and previous versions (know nothing about later ones).
Quote:
If you just google "why you don't need to defragment linux" you will get 216,000 hits like this one:
Most of them telling you the wrong argument I assume, like the one you linked. I stumped with that same thread long ago and posted a correction below, signed as "Jesgue" so you can search for that in the thread to see an extended explanation about elevators or i/o schedulers.
Being that said, you usually don't defragment linux. You could always back a partition up, then format it and restore the backup, which is like a poor man's defrag. That's not too appealing, I know.
Andy,
I don't generally defrag very often, but in the case of these files, I wanted the fastest access possible. According to the Win7 tools, none of my drives exceed 3% fragmentation, the 4TB drives < 1%.
My main objective with this thread was to rant about my stupidity in the initial copy files fiasco, but Dave supplied me with a useful hint that I will follow up on, i.e. "Are these Seagate 4 TB drives safe to connect to my old XP system"?
Dave.
Quote from: dedndave on September 18, 2013, 05:44:40 PM
being as the drives are > ~2 tb, they are GUID Partition Table (GPT) formatted, rather than MBR
XP doesn't support GPT directly, although there are drivers available
but, that may explain the defrag issue
i will give you a heads-up....
don't connect a GPT drive to an XP system without GPT support
it is possible that you lose a bunch of data
and, you may have trouble getting it to be a 4 tb drive, again :redface:
Dave,
You were right, only supported on XP SP3 according to the Website (the packaging only said XP).
Dave.
that's odd
i run XP SP3, and i had to install a driver to support GPT drives
perhaps what they mean is - they provide a driver, and it requires XP SP3 ?
Dave.
I think that is what they mean.
I have no need to do this, I already have several pairs of "too small" backup drives (750 GB) that will work quite well on the XP system.
Dave.
I am looking into downloading and using UltraDefrag.
Anyone know anything about it?
Dave.
i use Puran Defrag
that's on a 32-bit system
i am very happy with it
Quote from: KeepingRealBusy on September 21, 2013, 05:10:31 AM
Maybe I have to study the code some more, to see if they somehow can create a full sized file before copying the content. Maybe by creating the file, then seeking to the last BYTE, and writing a 0, then seeking to the front and starting the copy. Hard to follow the CPP code, especially with the unreadable "commentary".
SetFilePointer and SetEndOfFile maybe?
Also, the though,
Dave, maybe the slowdown while copying caused by Explorer files list in a window of a destination? Did you try to close the window of a folder where you targeted files to? I.e., when there are much files copied, Explorer gets more memory to save the listview where the files are displayed, as well as it constantly updates the information about files presented in the window. When there are much files, Explorer may probably take much resources for that target (memory loading and disk requests).
Quote from: dedndave on September 21, 2013, 12:52:10 PM
i use Puran Defrag
that's on a 32-bit system
i am very happy with it
Dave,
Did you get the freeware version?
Dave.
Quote from: Antariy on September 21, 2013, 01:03:22 PM
Quote from: KeepingRealBusy on September 21, 2013, 05:10:31 AM
Maybe I have to study the code some more, to see if they somehow can create a full sized file before copying the content. Maybe by creating the file, then seeking to the last BYTE, and writing a 0, then seeking to the front and starting the copy. Hard to follow the CPP code, especially with the unreadable "commentary".
SetFilePointer and SetEndOfFile maybe?
Also, the though, Dave, maybe the slowdown while copying caused by Explorer files list in a window of a destination? Did you try to close the window of a folder where you targeted files to? I.e., when there are much files copied, Explorer gets more memory to save the listview where the files are displayed, as well as it constantly updates the information about files presented in the window. When there are much files, Explorer may probably take much resources for that target (memory loading and disk requests).
Alex,
I don't think that that is the reason. The same sized files run executed in about 3.5 hours with 35 MBS transfers, it was the variable sized files copy that ran slowly with 16 MBS transfers, and this was with the same boot, the same ports, the same drives, the same displays.
Dave.
yes - the free version works nicely
The Big XP Defragmenter Test (http://www.hofmannc.de/en/windows-xp-defragmenter-test/benchmarks.html)
UniExtract can extract the contents of Puran tools. Some of the applications can be made portable.
http://www.winpenpack.com/en/download.php?view.1271
deleted
Hi nidud,
thank you for uploading the files and sources. By the way, it occurs another false positive generated by Avira.
Gunther
deleted
Quote from: jj2007 on September 21, 2013, 05:00:39 PM
The Big XP Defragmenter Test (http://www.hofmannc.de/en/windows-xp-defragmenter-test/benchmarks.html)
JJ,
Thank you very much for the very interesting test documentation. All I can say is WOW! Someone sure put a lot of work into that test.
Dave.
Hi nidud,
Quote from: nidud on September 22, 2013, 12:24:49 AM
Does it give any specific message?
I use JWLINK so maybe that is the problem?
If so, this file (http://masm32.com/board/index.php?topic=2344.msg24846#msg24846) should also give an error
no, keymap.exe doesn't generate virus alarm. Strange.
Gunther
deleted
Hi nidud,
dz.exe doesn't lead to an alert. But I think it's the silly AV.
Gunther
deleted