Revise concurrency

[Martoon-00]

Clarification and motivation

Let's take a thorough look at how concurrency is used here.

Acceptance criteria

• Our current use of concurrency is revised.
• Suggestions are made.
• Each suggested point, if got an approval, is implemented.

[Martoon-00]

Some my thoughts.

Use paralellization

We should make sure, that all actions for which parallelization makes sense - are applied in parallel.

Some examples:

• After Avoid resource leaking in scanners #220, looks like calling par to run all files parsing in parallel will never be a loss.

Do not overuse IO-based concurrency

Currently we verify all the links with a variation of mapConcurrently.

Given that local links checking is a pure action, we could use something lighter for them.
But this is debatable, we will have to try and see how much do we win in practice and whether is the improvement worth the increased complexity.

Limits

If an application can exploit many cores, usually it does not consume all the cores by default and the application provides -j flag for increasing the number of used OS threads.

However, in this case this is probably redundant? Even if the repository contains like 100 .md files, the overall amount of computations should not be larger than several seconds. If we exploit all the cores, that should be a large load for a very short duration.

And actually, I/O likely will dilute the computation part.