number of files/directories not shown correctly

[benglitz]

It would be more intuitive, if repocli first collected the count of all files/total amount of data to be transferred, and then correspondingly labelled the progressbar. Currently, one has little prediction of how long the upload will take before finishing.

[hurngchunlee]

Currently the total data size and number of files are counted when making directory walk through at the transfer source. Files in the visited (sub-)directories are put into a queue (i.e. GO channel) and carried out by four worker threads to process the transfer immediately.

The reason of such implementation is to avoid additional directory walk through just for counting the data size and number of files to be transferred, and to have the transfer takes place without waiting until the end of the counting.

However, because the queue is unbuffered, only up to four sub-directories can be visited at the time, which blocks the update of the counter. I believe it is the reason causing the progress bar looks rather unpredictable.

As a workaround, I have changed to use the buffered channel (with buffer size of 1M files). If there are less than 1M files involved in the transfer, this should make the counter updated faster while still doesn't need an extra directory walk to get the total data size and number of files.

[hurngchunlee]

@benglitz I wonder if you see improvement with this workaround.

[benglitz]

Yes, it has definitely improved! Thanks for implementing the workaround.

[hurngchunlee]

Thanks @benglitz I am closing this issue.