Rejected jobs are still displayed as 'Queued' with gem 'sidekiq-status' on /sidekiq/statuses

[jamesst20]

** Gems (all latest) **

gem 'sidekiq'
gem 'sidekiq-cron', '~> 1.2'
gem 'sidekiq-status'
gem 'sidekiq-unique-jobs', '~> 7.0.0.beta28'

** Description **
When I queue multiple jobs and that a lock rejects them, they are removed from the queue and added to processed as expected.

However, when visiting /sidekiq/statuses which is provided by the gem sidekiq-status, they are still marked as 'queued' excepted for the jobs that are not discarded.

** Expected behaviors **

Set them as completed or remove them

** Current behavior **

Show as queued while they are no longer queued.

[mhenrixon]

I have my own page for viewing statuses. You can find a Locks tab in the Sidekiq Web interface.

I know nothing about Sidekiq::Status so the fastest way to get my help on this would be to give me a failing test case or example where it goes wrong.

The more details the better. Otherwise, I can't really commit to fixing it.

Late last year, someone explained really nicely what went wrong with Sidekiq::Apartment and I was able to quickly create a simple fix for their problem.

[jamesst20]

Thanks for the fast reply, I can't seem to find the Locks tab you are talking, I will try to look deeper in the documentation.

** Setup **

1. Install Sidekiq, Sidekiq Unique Jobs, Sidekiq Status
2. Setup routing:

require 'sidekiq/web'
require 'sidekiq-status/web'

Rails.application.routes.draw do
 mount Sidekiq::Web => '/sidekiq'
end

3. The route /sidekiq/statuses should be working.

** Reproducing error **

Have a worker:

class MyWorker
  include Sidekiq::Worker
  include Sidekiq::Status::Worker
  
  sidekiq_options(
    lock: :until_executed,
    lock_args_method: ->(args) { [args.first] },
    on_conflict: :reject
  )

  def perform(name)
    puts "Running job #{name}..."
    at(50)
    sleep 10
    at(100)
    puts "Done..."
  end
end

1. Run Rails Server but not Sidekiq
2. Queue MyWorker a few times (let's say 3 times)
3. Run sidekiq
4. Visit /sidekiq/statuses and notice how there is 2 jobs queued and one completed

[mhenrixon]

First of all, you might want to add the web extension for this gem as well:

require 'sidekiq/web'
require 'sidekiq-status/web'
require 'sidekiq_unique_jobs/web'

Rails.application.routes.draw do
 mount Sidekiq::Web => '/sidekiq'
end

[mhenrixon]

From my investigation, this is due to the order of the middleware. To avoid confusion and teach people that the order of middleware is super important, I decided to not add the middleware automatically.

This part deserves to be clear and up to the user. If you try the master branch (or clear your middleware first) you should be able to set everything up and verify locally.

I added more information about this to the readme.

[jamesst20]

@mhenrixon

Thanks for the fast reply / fixes. I updated to 7.0.0 (non-beta).

I had never heard of the web extension you told me to add in routes. Can't seem to find anything about it. Also, it's giving me this error when I try to boot up the server:

/Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `**require': cannot load such file -- sidekiq-unique-jobs/web (LoadError)**
	from /Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
	from /Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:89:in `register'
	from /Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
	from /Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:44:in `require'

So in the mean time, I removed the require 'sidekiq-unique-jobs/web'

I updated my middleware chain to this

Sidekiq.client_middleware do |chain|
  chain.add SidekiqUniqueJobs::Middleware::Client
  chain.add Sidekiq::Status::ClientMiddleware, expiration: 12.hours
end

Sidekiq.server_middleware do |chain|
  chain.add SidekiqUniqueJobs::Middleware::Server
  chain.add Sidekiq::Status::ServerMiddleware, expiration: 12.hours
end

but I am still getting the same behavior.

Sidekiq v6.1.2
sidekiq-unique-jobs v7.0.0 (stable)

[jamesst20]

Update, looks like doing the opposite inside middleware worked !

Sidekiq.client_middleware do |chain|
  chain.add Sidekiq::Status::ClientMiddleware, expiration: 12.hours
  chain.add SidekiqUniqueJobs::Middleware::Client
end

Sidekiq.server_middleware do |chain|
  chain.add Sidekiq::Status::ServerMiddleware, expiration: 12.hours
  chain.add SidekiqUniqueJobs::Middleware::Server
end

Sidekiq turned off, I queued 4 workers that sleeps for 5 seconds. I ran sidekiq and I got instantly this in the ui:

5 seconds later:

Looks like it's all good :) !

[mhenrixon]

Yeah sorry, I sloppy pasted the wrong code. There is a section in the README that has been corrected.

[mhenrixon]

@jamesst20 require 'sidekiq_unique_jobs/web'

[ArturT]

Hi @mhenrixon

I believe the code example from readme is invalid. https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/README.md#sidekiq-status
I tried it and it left multiple jobs with queued status in sidekiq-status web UI (the same as on screenshot in the first post).

I noticed you said in readme something opposite to the code presented: The status middleware should run after uniqueness on client and before on server. .
So I reverted the order of middlewares and now it works fine for me.

Here is my working config:

Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
  end

  config.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes # default
    chain.add SidekiqUniqueJobs::Middleware::Server
  end

  SidekiqUniqueJobs::Server.configure(config)
end


Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
  end
end

[mhenrixon]

Dangit @ArturT! I knew I rushed that a bit much. I'll copy your example to the README.md, thanks a bunch for letting me know!

[mhenrixon]

Ok, I updated the README again 😶