Add basic load balancing

[belugum]

Count the number of requests in queue in each thread and use the thread with the smallest queue for new requests.
Related to issue #258, may fix issue #182.
This will avoid ignoring some requests when a thread is stuck.

[crow-clang-format]

--- include/crow/http_connection.h	(before formatting)
+++ include/crow/http_connection.h	(after formatting)
@@ -187,7 +187,7 @@
           std::function<std::string()>& get_cached_date_str_f,
           detail::task_timer& task_timer,
           typename Adaptor::context* adaptor_ctx_,
-          int &queue_length_):
+          int& queue_length_):
           adaptor_(io_service, adaptor_ctx_),
           handler_(handler),
           parser_(this),
@@ -705,7 +705,7 @@
 
         size_t res_stream_threshold_;
 
-        int &queue_length;
+        int& queue_length;
     };
 
 } // namespace crow
--- include/crow/http_server.h	(before formatting)
+++ include/crow/http_server.h	(after formatting)
@@ -196,7 +196,7 @@
             asio::io_service& is = *io_service_pool_[service_idx];
             auto p = new Connection<Adaptor, Handler, Middlewares...>(
               is, handler_, server_name_, middlewares_,
-                get_cached_date_str_pool_[service_idx], *task_timer_pool_[service_idx], adaptor_ctx_, task_queue_length_pool_[service_idx]);
+              get_cached_date_str_pool_[service_idx], *task_timer_pool_[service_idx], adaptor_ctx_, task_queue_length_pool_[service_idx]);
             task_queue_length_pool_[service_idx] += 1;
             acceptor_.async_accept(
               p->socket(),

[The-EDev]

Thanks a lot for your work, I have a few points aside from the formatting. Please take a look at them.

One more thing I noticed is the fact that task_queue_length_pool_[service_idx] could be accessed / edited by multiple threads, meaning it would be wise to make them atomic.

[The-EDev]

The code seems okay, with the exception of the comment below and the unit tests failing to compile.

Please let me know if you need to restart CI

[belugum]

Yes, I wasn't familliar with atomics so it didn't compile. It's fixed now :)

[The-EDev]

Seems alright, I'll run one final test myself to check that everything is good and approve/merge the PR.

Thanks again for the work!

[The-EDev]

I've checked the code once more and found a couple of interesting things (Along with a significant fault of my own that I need to fix ASAP)

[The-EDev]

The logic is sound and works fine as far as I can tell. I would just like to add a couple things to tie up any potential loose ends. What do you think of them @belugum? I promise this is the last review (I wanted to be excessively thorough because this is a vital part of the framework)

[crow-clang-format]

--- include/crow/http_server.h	(before formatting)
+++ include/crow/http_server.h	(after formatting)
@@ -195,7 +195,7 @@
             uint16_t service_idx = pick_io_service_idx();
             asio::io_service& is = *io_service_pool_[service_idx];
             task_queue_length_pool_[service_idx]++;
-            CROW_LOG_DEBUG << &is << " {" <<  service_idx << "} queue length: " << task_queue_length_pool_[service_idx];
+            CROW_LOG_DEBUG << &is << " {" << service_idx << "} queue length: " << task_queue_length_pool_[service_idx];
 
             auto p = new Connection<Adaptor, Handler, Middlewares...>(
               is, handler_, server_name_, middlewares_,
@@ -214,7 +214,7 @@
                   else
                   {
                       task_queue_length_pool_[service_idx]--;
-                      CROW_LOG_DEBUG << &is << " {" <<  service_idx << "} queue length: " << task_queue_length_pool_[service_idx];
+                      CROW_LOG_DEBUG << &is << " {" << service_idx << "} queue length: " << task_queue_length_pool_[service_idx];
                       delete p;
                   }
                   do_accept();

[belugum]

Looks good. I applied your suggestions, thanks!

[crow-clang-format]

--- include/crow/http_server.h	(before formatting)
+++ include/crow/http_server.h	(after formatting)
@@ -196,7 +196,7 @@
             uint16_t service_idx = pick_io_service_idx();
             asio::io_service& is = *io_service_pool_[service_idx];
             task_queue_length_pool_[service_idx]++;
-            CROW_LOG_DEBUG << &is << " {" <<  service_idx << "} queue length: " << task_queue_length_pool_[service_idx];
+            CROW_LOG_DEBUG << &is << " {" << service_idx << "} queue length: " << task_queue_length_pool_[service_idx];
 
             auto p = new Connection<Adaptor, Handler, Middlewares...>(
               is, handler_, server_name_, middlewares_,
@@ -215,7 +215,7 @@
                   else
                   {
                       task_queue_length_pool_[service_idx]--;
-                      CROW_LOG_DEBUG << &is << " {" <<  service_idx << "} queue length: " << task_queue_length_pool_[service_idx];
+                      CROW_LOG_DEBUG << &is << " {" << service_idx << "} queue length: " << task_queue_length_pool_[service_idx];
                       delete p;
                   }
                   do_accept();

[The-EDev]

Looks all good, Thank you very much for your work!

[belugum]

No problem, thanks for merging!