Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make WorkerPools and Queues flushable #10001

Merged
merged 12 commits into from
Jan 29, 2020
Merged

Make WorkerPools and Queues flushable #10001

merged 12 commits into from
Jan 29, 2020

Conversation

zeripath
Copy link
Contributor

@zeripath zeripath commented Jan 26, 2020 •
edited
Loading

  • Adds Flush methods to Queues and the WorkerPool
  • Further abstracts the WorkerPool
  • Adds a final step to Flush the queues in the defer from PrintCurrentTest
  • Fixes an issue with Settings inheritance in queues

This is a partial breakout from #9856

Signed-off-by: Andrew Thornton [email protected]

@zeripath zeripath added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Jan 26, 2020
@zeripath zeripath added this to the 1.12.0 milestone Jan 26, 2020
@zeripath zeripath changed the title Make WorkerPools and Queues flushable Queue: Make WorkerPools and Queues flushable Jan 26, 2020
@codecov-io
Copy link

codecov-io commented Jan 26, 2020 •
edited
Loading

Codecov Report

Merging #10001 into master will decrease coverage by <.01%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #10001      +/-   ##
==========================================
- Coverage   42.28%   42.27%   -0.01%     
==========================================
  Files         616      616              
  Lines       80742    80742              
==========================================
- Hits        34140    34132       -8     
- Misses      42402    42409       +7     
- Partials     4200     4201       +1
Impacted Files Coverage Δ
models/unit.go 37.03% <0%> (-2.47%) ⬇️
services/pull/patch.go 66.03% <0%> (-1.89%) ⬇️
modules/queue/manager.go 60.51% <0%> (-1.54%) ⬇️
modules/queue/workerpool.go 45.55% <0%> (-1.07%) ⬇️
routers/repo/view.go 38.59% <0%> (-0.88%) ⬇️
models/repo.go 49.66% <0%> (-0.14%) ⬇️
models/error.go 31.31% <0%> (+0.54%) ⬆️
services/pull/check.go 58.74% <0%> (+4.19%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c5faad...0848fc2. Read the comment docs.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jan 26, 2020
This was referenced Jan 26, 2020
Merged
Closed
guillep2k
guillep2k reviewed Jan 27, 2020
View reviewed changes
Copy link
Member

@guillep2k guillep2k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just scratched the surface. I'l pick this up again tomorrow.

integrations/testlogger.go Outdated Show resolved Hide resolved
integrations/testlogger.go Outdated Show resolved Hide resolved
6543
6543 approved these changes Jan 27, 2020
View reviewed changes
Copy link
Member

@6543 6543 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok ...

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jan 27, 2020
@zeripath
Make WorkerPools and Queues flushable
de0c994 
Adds Flush methods to Queues and the WorkerPool
Further abstracts the WorkerPool
Adds a final step to Flush the queues in the defer from PrintCurrentTest
Fixes an issue with Settings inheritance in queues

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath
Change to for loop
5da04b3
@zeripath
Add IsEmpty and begin just making the queues composed WorkerPools
74a30fc
@zeripath
subsume workerpool into the queues and create a flushable interface
1f031cb
@zeripath
Add manager command
a883a9a
@zeripath
Move flushall to queue.Manager and add to testlogger
b453c15
guillep2k
guillep2k reviewed Jan 28, 2020
View reviewed changes
Copy link
Member

@guillep2k guillep2k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really appreciate all the comments you've added! ☺️

modules/graceful/manager_unix.go Outdated Show resolved Hide resolved
modules/graceful/manager_windows.go Show resolved Hide resolved
modules/queue/manager.go Show resolved Hide resolved
modules/queue/manager.go Show resolved Hide resolved
modules/queue/queue_wrapped.go
q.lock.Lock()
if q.internal == nil {
q.lock.Unlock()
return fmt.Errorf("not ready to flush wrapped queue %s yet", q.Name())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this really return an error? A hairy problem indeed.... 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the queue isn't flushed - so I think it should... This function isn't actually wired into anything though as wrapped queues don't actually implement Flushable - (they almost do).

I probably need to rethink this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's OK to return an error under the following light: the admin wants to know that everything was processed, nothing is pending, but the worker is still initializing, so guess what? I don't know if anything is pending, hence I return an error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that's the interpretation I mean.

modules/queue/queue_wrapped.go Outdated Show resolved Hide resolved
modules/queue/queue_wrapped.go Outdated Show resolved Hide resolved
routers/private/manager.go Show resolved Hide resolved
guillep2k
guillep2k approved these changes Jan 28, 2020
View reviewed changes
Copy link
Member

@guillep2k guillep2k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job! 🎉

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Jan 28, 2020
6543
6543 requested changes Jan 28, 2020
View reviewed changes
Copy link
Member

@6543 6543 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just looking throu

@techknowlogick techknowlogick merged commit c01221e into go-gitea:master Jan 29, 2020
@zeripath zeripath deleted the flush-queues branch January 29, 2020 04:52
@zeripath zeripath mentioned this pull request Feb 10, 2020
Merged
@zeripath zeripath changed the title Queue: Make WorkerPools and Queues flushable Make WorkerPools and Queues flushable May 17, 2020
@zeripath zeripath mentioned this pull request May 17, 2020
Merged
@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@guillep2k guillep2k guillep2k approved these changes

@6543 6543 6543 approved these changes

Assignees
No one assigned
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/feature Completely new functionality. Can only be merged if feature freeze is not active.
Projects
None yet
Milestone
1.12.0
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@zeripath @codecov-io @guillep2k @6543 @techknowlogick @GiteaBot @lafriks