Author Topic: A question for anyone with a Linux background.  (Read 406 times)

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 7460
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
A question for anyone with a Linux background.
« on: May 16, 2020, 03:40:06 PM »
I have set up a Linux box for a task of copying data from one disk to another. Version is Mint 64 bit 19.3 and it is up and going OK. I put data on one hard disk and tried to copy it to another hard disk that are both mounted as NTFS but when I used the standard file manager in Mint, it was so slow that I ended up turning it off.

I wondered if anyone knows enough about Linux file managers to recommend a good way to copy files from one NTFS partition to another ? I don't want disk image copy, just file copy in much the same way as I can do in the old Windows file manager or explorer.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :skrewy:

jj2007

  • Member
  • *****
  • Posts: 10464
  • Assembler is fun ;-)
    • MasmBasic
Re: A question for anyone with a Linux background.
« Reply #1 on: May 16, 2020, 07:12:35 PM »
Have you tried zipping, copy one archive, unzip? It might speed up the process...

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 7460
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: A question for anyone with a Linux background.
« Reply #2 on: May 16, 2020, 07:38:03 PM »
I am testing out an app called Xfe that is surprisingly good to navigate around a Linux distribution and while its no rocket, it is at least copying the files at a reasonable rate. Its the first file manager style app in Linux that I actually like. Your suggestion is probably faster but its more complicated to set up, I want this box among other things to copy data from one disk to another in much the same way as Windows does it and so far its OK.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :skrewy:

Vortex

  • Member
  • *****
  • Posts: 2283
Re: A question for anyone with a Linux background.
« Reply #3 on: May 16, 2020, 07:51:22 PM »
Hi Hutch,

Could you try the command line tool rsync to copy the files?

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 7460
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: A question for anyone with a Linux background.
« Reply #4 on: May 16, 2020, 08:53:15 PM »
I will give it a whirl once I know how it works. I just had the most unusual problem, when I came back to see if the file copy was finished, the monitor was dead. I changed the monitor and it still did not work so I changed the power cable and off it went. I have never seen that before.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :skrewy:

Vortex

  • Member
  • *****
  • Posts: 2283
Re: A question for anyone with a Linux background.
« Reply #5 on: May 16, 2020, 09:49:00 PM »
Hi Hutch,

Here is an example :

/mnt/ntfs1/folder1 : source
/mnt/ntfs2/folder2/ : destination

Code: [Select]
rsync -avz /mnt/ntfs1/folder1 /mnt/ntfs2/folder2/

mineiro

  • Member
  • ****
  • Posts: 592
Re: A question for anyone with a Linux background.
« Reply #6 on: May 16, 2020, 09:52:50 PM »
I did some tests and the result was unsatisfactory. From NTFS to ext4 it was about 3 times faster than NTFS to NTFS.
I did the test with the cp command and the same result was obtained. To have a margin on the transfer I used the dd command with caution that copies only 1 file.
A search led me to see the command mount and ntfs-3g. Some said to remove the "sync" key (/etc/fstab) from the assembly line while others said to insert the "-o big_writes" key. I honestly saw no difference. I tested other keys like async and noatime.
One plausible explanation I found was about the switch from kernel mode to user mode. In the website:
https://www.tuxera.com/community/ntfs-3g-faq/#slow
"Status: An optimized, high-performance version, primarily for consumer electronics devices, is available commercially."
An unknown question is when I see the assembly command line I am not shown the big_writes key. Perhaps it has become standard.
I don't see any gain!!!. Maybe solution is find other driver that deals with NTFS in linux.

If you want to try, there are two possible ways:
1- view/edit the /etc/fstab file as root and insert the big_writes key (async,noatime) as shown below. Changes to this file will affect the next boot or mount:
$ cat /etc/fstab
Code: [Select]
...
#Entry for /dev/sda1 :
UUID=3C382BD6382B8DC8   /media/3C382BD6382B8DC8 ntfs    defaults,nosuid,nodev,big_writes,uhelper=udisks,uid=1000,gid=1000,dmask=0077,fmask=0177 0       0
#Entry for /dev/sda5 :
UUID=FAA073C9A0738AC1   /media/FAA073C9A0738AC1 ntfs    defaults,nosuid,nodev,big_writes,uhelper=udisks,uid=1000,gid=1000,dmask=0077,fmask=0177 0       0

2- Viewing partitions/mounted drives, dismounting and mounting only in this instance, the keyword to NTFS is fuseblk. Or using the df command
Code: [Select]
$ mount
...
/dev/sdb5 on /media/FAA073C9A0738AC1 type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
/dev/sdb1 on /media/3C382BD6382B8DC8 type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
$ sudo umount /media/FAA073C9A0738AC1
$ sudo umount /media/3C382BD6382B8DC8
if you type only mount again you will not see then mounted.
$ sudo mount -o big_writes /dev/sdb5 /media/FAA073C9A0738AC1/
$ sudo mount -o big_writes /dev/sdb1 /media/3C382BD6382B8DC8/

Code: [Select]
------tests - dd is dangerous, use with caution, only one file operation, not directories or folders--------
$ dd if=/media/3C382BD6382B8DC8/bkp.pdf of=/media/FAA073C9A0738AC1/bkp.pdf
32644 + 1 check-ins
32644 + 1 check-out
16714046 bytes (17 MB) copied, 0.837292 s, 20.0 MB / s

After umounting and mounting, the results are not good:
$ sudo umount /media/FAA073C9A0738AC1
$ sudo umount /media/3C382BD6382B8DC8
$ sudo mount -o big_writes /dev/sdb5 /media/FAA073C9A0738AC1/
$ sudo mount -o big_writes /dev/sdb1 /media/3C382BD6382B8DC8/
$ dd if=/media/3C382BD6382B8DC8/bkp.pdf of=/media/FAA073C9A0738AC1/bkp1.pdf
32644 + 1 check-ins
32644 + 1 check-out
16714046 bytes (17 MB) copied, 1.00576 s, 16.6 MB / s

$ sudo umount /media/FAA073C9A0738AC1
$ sudo umount /media/3C382BD6382B8DC8
$ sudo mount -o big_writes -o noatime -o async /dev/sdb1 /media/3C382BD6382B8DC8/
$ sudo mount -o big_writes -o noatime -o async /dev/sdb5 /media/FAA073C9A0738AC1/
$ dd if=/media/3C382BD6382B8DC8/bkp.pdf of=/media/FAA073C9A0738AC1/bkp2.pdf
32644 + 1 check-ins
32644 + 1 check-out
16714046 bytes (17 MB) copied, 0.851619 s, 19.6 MB / s
----
I don't have tried rsync to copy files as sugested by sir Vortex.
I'd rather be this ambulant metamorphosis than to have that old opinion about everything

mineiro

  • Member
  • ****
  • Posts: 592
Re: A question for anyone with a Linux background.
« Reply #7 on: May 16, 2020, 11:19:21 PM »
If you have problems with these command lines, just post output of mount and/or /etc/fstab file and I can create a command line to be applied.
In shell, you can type <TAB> to autocomplete names. You can select shell strings by pressing <SHIFT> key and doing the rest with mouse. To copy you can <CTRL+SHIFT+C>. To paste in user interface you can use <CTRL+V> while in console/terminal mode you can paste with <CTRL+SHIFT+V>. You can press ↑↓ keys in keyboard to activate last typed commands (like doskey in ms-dos).
Command df shows disk free space in mounted partitions/volume.
Command dd can clone disks, thats why I told to be caution. Can be used to copy only one file.
dd if=/path/inputfile of=/path/outputfile

from my example yours can differs in sd??, like sda1,sda5,sdb1, ... ;  differs from mounting folder too (/media/FAA073C9A0738AC1,/media/3C382BD6382B8DC8), and can differ UUID=????? in fstab file.
I'd rather be this ambulant metamorphosis than to have that old opinion about everything

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 7460
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: A question for anyone with a Linux background.
« Reply #8 on: May 17, 2020, 12:20:03 AM »
I will get there, I am just a bit brain dead at the moment, finished a messy and complex piece of code today for doing the settings for scaling images in SSE2,  I tried out WINE but it was very poor so I removed it, got more of the settings done, mounted partitions are now persistent, copied about 1.3 gig of screens as I like mine more than theirs, installed Google Earth and my preferred Slimjet browser.

Interestingly enough, I copied the screens onto a fast USB drive and loaded them into the "pictures" directory in my home profile and the transfer was genuinely fast so I have probably hit a winner with the file manager. Can find most of the useful utilities on the menu so I am getting there.

I am OK with command lines, many years practice but finding everything and identifying what does what is what takes the time but it looks like it will do the job at the moment and it may be useful on the side. I already have a Linux Mint file server which I mainly use as dead storage that is full of my old XP era 2tb disks and with this next one, I have not set up the Samba server as it is not really pointed at that task.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :skrewy:

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 7460
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: A question for anyone with a Linux background.
« Reply #9 on: May 19, 2020, 09:01:39 PM »
After testing out an old Intel desktop board which works OK with a new CMOS battery, I have done what I hope is the last rebuild, I used a Core2 quad 9650 on a later Intel desktop board, stuck it all in a new can that was not suitable for other tasks and installed Linux Mint 19.3 and it all works OK. I have aspirations of later putting another board into this can but at the moment its up and running and works well. Linux Mint actually worked well on the older board, a 2.4 gig Core2 quad with 4 gig of memory (an old XP box board).

The main win is it has 2 removable HDDs so I can use it to copy data from one disk to another which is far faster than a gigabit network.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :skrewy:

carlos

  • Member
  • **
  • Posts: 71
Re: A question for anyone with a Linux background.
« Reply #10 on: July 12, 2020, 08:25:20 AM »
rsync is EXCELLENT for copy bunch of files, (even whole disks). but being CLI may seem hard on someone more accustomed to GUI ... so there is a GUI wrapper for rsyn. GRSYNC, and not only for Linux, it exist in windows also 

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 7460
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: A question for anyone with a Linux background.
« Reply #11 on: July 12, 2020, 08:31:13 AM »
 :biggrin:

I would not worry too much about that, I can still remember MS-DOS LINK.EXE command lines. None the less I have found a Linux file manager that seems to do this style of copy fast enough, where NEMO was very slow this new one performs well. I will have a look at rsync as it sounds like a useful tool.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :skrewy: