Sync performance

[renbrauge]

I've been looking at the performance of the sync operation, in my understanding, the operation spends time in two areas:

• diff2, server side bound, do you have any recommendation on how to make this faster? (better server cpu/disk? a configuration on the P4 server?)
• Creating the symlinks, which is client bound, maybe more parallelization would help on that end?

[jessk-msft]

Hi Ren, the diff2 command in DepotOperations::SyncCommand is an unfortunate necessity right now. I wish there was an alternative, but know of no other way around this.

The diff2 command is used to gather filetype modifications to file revisions between the #have and the target changelist. Since filetype changes do not appear in sync -n results, we have to issue this diff2 command to see if file attributes need to be changed, ie (readonly to writable, and vise-versa).

Please reach out if anyone knows another technique or perhaps additional options to receive this information from sync -n.
-Jess

[jspelletier]

Hi,
I think there might be a way to avoid p4 diff2 as the our internal similar tool at Ubisoft(pvfs) doesn't do any diff2. Here is an anonymized and truncated log for a sync from scratch done on my PC using that tool. It might give you some insight about how to solve this and accelerate commands.

# pvfs sync
[2023-06-19 13:35:17.160] Running p4 client -o -t jsp_pc_testpvfs jsp_pc_testpvfs
[2023-06-19 13:35:17.167] Reloading from jspelletier:somep4server:3806 at E:\1806\testpvfs...
[2023-06-19 13:35:17.167] Executing p4 parallel fstat...
[2023-06-19 13:35:17.168] Running p4 dirs -H //jsp_pc_testpvfs/*#have
[2023-06-19 13:35:17.169] Running p4 fstat -Ol -T clientFile,headRev,headType,action,fileSize,digest //jsp_pc_testpvfs/*#have
[2023-06-19 13:35:17.178] Running p4 fstat -Ol -Ro -T clientFile,fileSize,headRev,headType,digest,action //jsp_pc_testpvfs/...
[2023-06-19 13:35:17.186] //jsp_pc_testpvfs/... - file(s) not opened on this client.
[2023-06-19 13:35:17.187] GetFStatOpened //jsp_pc_testpvfs/... took (8.70 ms)
[2023-06-19 13:35:17.188] Save took (0.72 ms)
[2023-06-19 13:35:17.189] P4 fstat finished (took 0.02 s)
[2023-06-19 13:35:17.189] Loading p4 fstat data D41D8CD98F00B204E9800998ECF8427E...
[2023-06-19 13:35:17.190] P4 fstat data loaded (0 files, 0 valid) (took 0.12 ms)
[2023-06-19 13:35:17.191] Waiting for directory virtualization check...
[2023-06-19 13:35:17.191] Directory virtualization check finished (took 0.35 ms)...
[2023-06-19 13:35:17.192] Running reconcile on non-clobber workspace to checkout locally modified files before sync...
[2023-06-19 13:35:17.193] Running p4 client -o -t jsp_pc_testpvfs jsp_pc_testpvfs
[2023-06-19 13:35:17.199] Reconcile finished (took 0.01 s)
[2023-06-19 13:35:17.199] Getting last submitted changelist...
[2023-06-19 13:35:17.200] Running p4 changes -s submitted -m1
[2023-06-19 13:35:17.204] Last submitted changelist : 7607896
[2023-06-19 13:35:17.204] Modified sync arguments: @7607896
[2023-06-19 13:35:17.212] Running p4 sync -N @7607896
[2023-06-19 13:35:28.652] Syncing quietly to improve performance (204575 files are involved) ...
[2023-06-19 13:35:28.660] Running p4 sync -k -q @7607896
[2023-06-19 13:35:40.774] Sync finished (took 23.62 s)
[2023-06-19 13:35:40.774] Running p4 client -o -t jsp_pc_testpvfs jsp_pc_testpvfs
[2023-06-19 13:35:40.781] Reloading from jspelletier:somep4server:3806 at E:\1806\testpvfs...
[2023-06-19 13:35:40.781] Executing p4 parallel fstat...
[2023-06-19 13:35:40.782] Running p4 dirs -H //jsp_pc_testpvfs/*#have
[2023-06-19 13:35:40.782] Running p4 fstat -Ol -T clientFile,headRev,headType,action,fileSize,digest //jsp_pc_testpvfs/*#have
[2023-06-19 13:35:40.789] Running p4 dirs -H //jsp_pc_testpvfs/XXX/*#have
[2023-06-19 13:35:40.791] Running p4 dirs -H //jsp_pc_testpvfs/XYZ/*#have
[2023-06-19 13:35:40.795] Running p4 fstat -Ol -T clientFile,headRev,headType,action,fileSize,digest //jsp_pc_testpvfs/XXX/*#have
[2023-06-19 13:35:40.797] Running p4 fstat -Ol -T clientFile,headRev,headType,action,fileSize,digest //jsp_pc_testpvfs/XYZ/source/...#have
[2023-06-19 13:35:41.616] Running p4 fstat -Ol -Ro -T clientFile,fileSize,headRev,headType,digest,action //jsp_pc_testpvfs/...
[2023-06-19 13:35:41.624] //jsp_pc_testpvfs/... - file(s) not opened on this client.
[2023-06-19 13:35:41.624] GetFStatOpened //jsp_pc_testpvfs/... took (8.26 ms)
[2023-06-19 13:35:41.801] Save took (176.68 ms)
[2023-06-19 13:35:41.801] P4 fstat finished (took 1.02 s)
[2023-06-19 13:35:41.835] Workspace changed, updating workspace from D41D8CD98F00B204E9800998ECF8427E to EED52577E869C87C47F857587BB2D62B
[2023-06-19 13:35:41.978] Workspace files updated: 0
[2023-06-19 13:35:41.978] Workspace files deleted: 0
[2023-06-19 13:35:41.979] Workspace update complete (took 0.14 s)

[jessk-msft]

Hi jspelletier,
Thanks for these details about how your pvfs sync operation is performed. Perhaps it would be faster if the p4vfs sync gathered an fstat of the filespec at #have and at the target changelist, then compared them to make necessary filetype changes.