Thanks for this.
I did some checking, and Raj's suggestion seems to be confirmed, though there are some aspects of the reported
information that don't fully make sense, since both disks are USB external ones. The Seagate 3 Terabyte is a
3.5 inch with external power supply. The Western Digital is a 2.5" USB-powered one. The "mount" information
says Seagate is ext4, but fdisk thinks its dos. Controller circuitry in the disks for the USB perhaps?
Here's a summary.
From fdisk -l:
Disk /dev/sde: 2.7 TiB, 3000592977920 bytes, 732566645 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0002161c
Device Boot Start End Sectors Size Id Type
/dev/sde1 256 732566527 732566272 2.7T 83 Linux
Disk /dev/sdf: 2.7 TiB, 3000558944256 bytes, 5860466688 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: A54602B4-CC02-4A3C-A3BF-14BDCDB1C2F0
Device Start End Sectors Size Type
/dev/sdf1 2048 5860464639 5860462592 2.7T Microsoft basic data
john@john-j6-18 ~ $
From df:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sde1 2884152536 1876450388 861172512 69% /media/john/Seagate3T
/dev/sdf1 2930231292 1914905744 1015325548 66% /media/john/RedWD3T
john@john-j6-18 ~ $
From mount:
/dev/sde1 on /media/john/Seagate3T type ext4 (rw,nosuid,nodev,relatime,data=ordered,uhelper=udisks2)
/dev/sdf1 on /media/john/RedWD3T type fuseblk
(rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks2)
Best, JN
On 2019-01-07 9:21 a.m., Raj wrote:
> Ah that makes sense. I can (possibly) explain this behaviour, I have seen it before. By default, rsync will compare
> filesize and timestamp before deciding to proceed with a full file compare. Somehow your incarnation seems to think the
> size and/or timestamp differs, and is moving on to a block-by-block compare, which is what you see in the strace. The
> block by block is slow and computationally intense, so it makes sense to avoid it if possible.
>
> The times I have seen rsync drop to block by block even though the file hasn't changed is because one of the filesystems
> does not store timestamps to the same precision as the other (smb shares), or one of the filesystems has a different
> blocksize than the other, which always throws the size comparison off. If you have one of these, or something similar,
> check out the -c flag - it will compare the file using a checksum rather than timestamp or size. While it will be
> slower than the default check, it will be a lot faster than a block by block compare for files that haven't changed
> since the last rsync.
>
> cheers!
> Raj.
>
> On Mon, Jan 7, 2019 at 9:13 AM J C Nash <profjcnash [ at ] gmail [ dot ] com <mailto:profjcnash [ at ] gmail [ dot ] com>> wrote:
>
> My script uses the -auv flags, so I see stuff going on. It was when this stopped
> for a very long time at a point where I did not expect that I though it had hung.
>
> I ran things again, though from a terminal, and when it seemed to hang ran
> strace -p <number>
> and could see things going on (no copy, just check and compare). And eventually it
> terminated normally after an hour or so. Actually I went to bed, but could see steady
> progression through the directory tree, since my backups are dated by year and month.
>
> Why I did not see this in the rsync script output I don't know (my Double Commander
> icon opens a terminal -- I like to have progress indicated). Have not
> nailed down precisely why strace shows activity but rsync does not, and suspect
> I'll need to delve into details of the code (I may end up having to give a talk),
> but it seems that rsync is working. It is also possible, as suggested by (I think)
> Alex, that the disk or controllers could be flakey, though the target disk is only
> a few months old. Both source and target are USB connected, which is another possible
> point to investigate.
>
> Thanks to all who responded so quickly. And I'll have to remember strace. Not a
> command I've used before.
>
> Best, JN
>
>
>
> On 2019-01-06 9:49 p.m., Richard Guy Briggs wrote:
> > On 2019-01-06 17:49, J C Nash wrote:
> >> The title says it. I have rsync set up to run from the Double Commander 2 pane file
> >> manager. If I select parallel directories of modest size, rsync runs fine. (There are
> >> very few or no updates. I'm just trying to ensure all is up to date.)
> >>
> >> However if I choose a high level directory on each side, it goes for a while then seems to
> >> just hang.
> >
> > Are you able to "strace -p <pid>" on that process (on each machine if over ssh)
> > to see if it is actually hung or doing something potentially useful?
> >
> >> I'm wondering if I've overflowed some sort of index or buffer. Suggestions?
> >>
> >> I don't mind if it gives me a warning and stops. Then I can simply use smaller chunks.
> >> However, no message and not stopping is a problem.
> >>
> >> Best, JN
> >
> > slainte mhath, RGB
> >
> > --
> > Richard Guy Briggs -- ~\ -- ~\ <hpv.tricolour.ca <http://hpv.tricolour.ca>>
> > <www.TriColour.ca <http://www.TriColour.ca>> -- \___ o \@ @ Ride yer bike!
> > Ottawa, ON, CANADA -- Lo_>__M__\\/\%__\\/\%
> > Vote! -- <greenparty.ca <http://greenparty.ca>>_____GTVS6#790__(*)__(*)________(*)(*)_________________
> >
>
> To unsubscribe send a blank message to linux+unsubscribe [ at ] linux-ottawa [ dot ] org <mailto:linux%2Bunsubscribe [ at ] linux-ottawa [ dot ] org>
> To get help send a blank message to linux+help [ at ] linux-ottawa [ dot ] org <mailto:linux%2Bhelp [ at ] linux-ottawa [ dot ] org>
> To visit the archives: https://lists.linux-ottawa.org
>
To unsubscribe send a blank message to linux+unsubscribe [ at ] linux-ottawa [ dot ] org
To get help send a blank message to linux+help [ at ] linux-ottawa [ dot ] org
To visit the archives: https://lists.linux-ottawa.org