Support for GNU Make jobserver (alternative implementation) #104
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Apr 19, 2024
Prepare for adding more "if ... continue" at the top. Signed-off-by: Paolo Bonzini <[email protected]>
|
(rebased on top of #106, which provides "give back tokens on signals" behavior for free) |
Keep the system clean by propagating SIGTERM to all children. The only tricky bit is that jobs[i].edge is used to detect jobs that are in use and therefore must be killed. Signed-off-by: Paolo Bonzini <[email protected]>
SIGINT has a different behavior than SIGTERM in GNU Make; do the same in Samurai. SIGINT only prevents new jobs from starting, but it will wait for the running ones to finish. Because SIGINT is sent to all processes in the group, they will stop on their own and there is no need to kill them. Signed-off-by: Paolo Bonzini <[email protected]>
GNU Make has a neat feature called the jobserver protocol, where the top-level Make can allocate a specific number of job slots, and child makes can take slots to do work in. This was designed to stop the parallelisation problem where a top-level make -j10 may potentially spawn 10 separate sub-makes all with -j10 so there's now 100 parallel jobs. However, it's also useful for resource control is systems which built multiple pieces of software at once. For example, Bitbake can build N different pieces of software at once, and each of those is passed a -jM flag. If each of these N tasks is compiling then thats's N*M jobs so you don't want N or M to be too high, but if only 1 of N is building then you want M to be high. With the job server protocol there are N slots in total for all sub makes, so you can control the resource utilisation more accurately. By supporting the jobserver protocol instead of just -j, Samurai can join in the resource pooling and builds can be more efficient. Signed-off-by: Paolo Bonzini <[email protected]>
Signed-off-by: Paolo Bonzini <[email protected]>
Signed-off-by: Paolo Bonzini <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Alternative implementation to #94.
The main advantage is that the integration with the
build()event loop is very clean, as it simply uses a pipe to signal the availability of tokens. Interacting with the job server is entirely embedded within a newtoken.cfile that implements a simple API:and on top of this, the integration is about 20 lines of code.
On the other hand
token.cuses pthreads, which perhaps could be considered less appealing. Waiting for reviews. :)