-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Progress bar #150
Comments
That would be great, Matt! Do you think this something that can be added to our Jar file https://github.com/ipeaGIT/r5r/blob/master/r-package/inst/jar/r5r_0_3_0.jar ? |
Thanks Matt. I had planned to add some simple progress info like printing "processing origin x out of y" in the console, but a progress bar would be much better. I actually don't know how to do this in Java without interfering with R5's outputs in the console. |
Great. I probably won't work on this for a month or so, but putting some implementation notes here so I remember them then:
|
Hi @mattwigway! Have you made any progress on this issue? |
…verbose = FALSE` (Issue #150). Small tweaks in the documentation.
I've added a quite simple implementation of this feature in the |
Thanks, @mvpsaraiva . I'm wondering if this could this be slowing r5r down, though. I had the impression it's slower after quick test |
Technically, it has to slow things down a bit, because it needs to synchronise the counter between all threads. I've ran it 3 times, because R5 gets faster over time. Here are the results:
|
Hm.... so showing the counter can make computation process 20% to 40% slower. This seems a lot. @mattwigway , any suggestions here? |
Yes, the synchronization is likely the problem - anything that requires
synchronization can be slow. You might try creating a ConcurrentLinkedQueue
and pushing an object onto it from the parallel stream that runs the
results, and then have another thread that pops things off that queue and
increments a result counter. I think inserting into the
ConcurrentLinkedQueue is probably still synchronized internally, but no IO
is happening in the synchronized block so it won't freeze the process for
as long.
Alternately, you could take a look at the LambdaCounter class in R5. It is
synchronized much like the code @mvpsaraiva wrote, but only logs at
specified intervals so most synchronized blocks don't require IO.
Matt
…On Fri, May 28, 2021 at 9:46 AM Rafael H M Pereira ***@***.***> wrote:
Hm.... so showing the counter can make computation process 20% to 40%
slower. This seems a lot. @mattwigway <https://github.com/mattwigway> ,
any suggestions here?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#150 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAEKNLWLKXNX5TGPTCRN2BDTP6NCZANCNFSM4YFCXZ4Q>
.
|
I've suspected the 20% to 40% slow down was too much, and probably due to the small test area (r5r's Porto Alegre sample dataset), then I've run some more tests on an entire city (Belo Horizonte). I've also looked at LambdaCounter as @mattwigway suggested, and implemented versions that show progress at intervals and as percents.
There are a few differences between the runs, but they're probably more due to other things than the progress method.I think it's probably fine to use any of them. I think the large overhead of the previous test was because, in a small study area, it's more likely that more threads finish computing travel times at once and try to update the counter simultaneously, causing a slow down. In larger study areas, threads are more likely to complete tasks asynchronously and access the counter at different times. |
Setting this issue as priority, to be incorporated in the next package release. |
Hi both. I've updated the documentation of the
|
A progress bar for the travel_time_matrix function would be very handy for larger runs, and not too difficult to implement (it would have to be on the Java side). There's some tricks with safe multithreading, but not too bad - I'm happy to take this on myself.
The text was updated successfully, but these errors were encountered: