Configure Cloud Tasks queues

This page describes how to configure Cloud Tasks queues using the gcloud command of the Google Cloud CLI.

Overview

You can configure your Cloud Tasks queue when you create the queue or anytime afterwards, and the configuration will apply to all tasks in that queue.

There are three basic aspects to configuring your queues:

Queue-level routing

Configuring routing at the queue level overrides routing set at the task level. This is useful if you want to use Cloud Tasks as a buffer in front of your target service, or if you need to change the routing for all tasks in a queue.

Queue-level routing applies to:

  • Tasks that are currently in the queue
  • Tasks that are added to the queue after queue-level routing has been set

Limitations

Queue-level routing is not compatible with Cloud Key Management Service (Cloud KMS) customer-managed encryption keys (CMEK). If CMEK is enabled, you cannot:

  • Create tasks on a queue that has queue-level routing
  • Apply queue-level routing

Configure queue-level routing for HTTP tasks

Apply queue-level routing

You can configure a queue to override task-level routing either when creating the queue or when updating the queue. To configure queue-level routing, set the queue's uriOverride parameter to your preferred route. If you are applying queue-level routing as an update to an existing queue, pause the queue before applying the changes and wait one minute after applying the changes to resume the queue. It can take up to one minute for the new configuration to take effect, so waiting to resume the queue helps to prevent tasks from dispatching with the old configuration.

Update or remove queue-level routing

  1. Pause the queue.

    Console

    To pause your queue using the Google Cloud console:

    1. Open the Cloud Tasks queues page in the console.

      Go to the Cloud Tasks queues page

    2. Select the name of the queue you want to pause and click Pause queue.

    3. Confirm the action.

    gcloud

    gcloud tasks queues pause QUEUE_ID

    Replace QUEUE_ID with the ID of your queue.

  2. Update or remove queue-level routing.

    • To update queue-level routing, set the uriOverride parameter to your updated route.

    • To remove queue-level routing using either the REST or RPC API:

      • REST API: Send a patch request for the queue with an empty payload and the updateMask parameter set to httpTarget.

      • RPC API: Send an updateQueueRequest for the queue with an empty payload and the update_mask parameter set to http_target.

    The following example uses the REST API to update the host tasks are routed to:

    curl -X PATCH -d @- -i \
-H "Authorization: Bearer ACCESS_TOKEN" \
-H "Content-Type: application/json" \
"https://cloudtasks.googleapis.com/v2/projects/PROJECT_ID/locations/LOCATION/queues/QUEUE_ID?updateMask=httpTarget.uriOverride" << EOF
{
"httpTarget": {"uriOverride":{"host":"NEW_HOST"}}
}
EOF

    Replace the following:

    • PROJECT_ID: The ID of your Google Cloud project. You can get this by running the following in your terminal: 
      gcloud config get-value project
    • LOCATION: The location of your queue.
    • QUEUE_ID: The ID of your queue.

    • ACCESS_TOKEN: Your access token. You can get this by running the following in your terminal:

      1. gcloud auth application-default login
      2. gcloud auth application-default print-access-token

    • NEW_HOST: The new host you want your queue to route to.

  3. Wait one minute.

    It can take up to one minute for the new configuration to take effect, so waiting to resume the queue helps to prevent tasks from dispatching with the old configuration.

  4. Resume the queue.

    Console

    To resume your queue using the Google Cloud console:

    1. Open the Cloud Tasks queues page in the console.

      Go to the Cloud Tasks queues page

    2. Select the name of the queue you want to pause and click Resume queue.

    3. Confirm the action.

    gcloud

    gcloud tasks queues resume QUEUE_ID

    Replace QUEUE_ID with the ID of your queue.

Configure queue-level routing for App Engine tasks

To configure queue-level routing for App Engine tasks, set the queue's appEngineRoutingOverride parameter to your preferred App Engine service and version.

To set up this non-default queue-level routing and override any task-level routing, you can use gcloud:

  gcloud tasks queues update QUEUE_ID\
      --routing-override=service:SERVICE,version:VERSION

Replace the following:

  • SERVICE: the App Engine worker service responsible for task handling.
  • VERSION: the app version.

For example, if you set up a worker service SERVICE to handle all tasks in a queue, you can route to that service and the default version:

  gcloud tasks queues update QUEUE_ID \
      --routing-override=service:SERVICE

Describe the queue:

    gcloud tasks queues describe QUEUE_ID --location=LOCATION

Replace the following:

  • QUEUE_ID: the queue ID (its short name)
  • LOCATION: the location of the queue

The output should be something like:

appEngineRoutingOverride:
  host: SERVICE.PROJECT_ID.appspot.com
  service: SERVICE
name: projects/PROJECT_ID/locations/LOCATION_ID/queues/QUEUE_ID
rateLimits:
  maxBurstSize: 100
  maxConcurrentDispatches: 1000
  maxDispatchesPerSecond: 500.0
retryConfig:
  maxAttempts: 100
  maxBackoff: 3600s
  maxDoublings: 16
  minBackoff: 0.100s
state: RUNNING

Remove queue-level routing for App Engine targets

When queue-level routing is removed, task-level routing applies both to tasks currently in the queue and tasks added to the queue in the future. To remove queue-level routing, run the following command:

  gcloud tasks queues update QUEUE_ID \
      --clear-routing-override

Define rate limits

You can set the maximum rate and number of concurrent tasks that can be dispatched by a queue.

  gcloud tasks queues update QUEUE_ID \
      --max-dispatches-per-second=DISPATCH_RATE \
      --max-concurrent-dispatches=MAX_RUNNING

Replace the following:

  • DISPATCH_RATE: the dispatch rate (this is the rate at which tokens in the bucket are refreshed). In conditions where there is a relatively steady flow of tasks, this is the equivalent of the rate at which tasks are dispatched.
  • MAX_RUNNING: the maximum number of tasks in the queue that can run at once.

For example, if you created a queue without setting any parameters, you can update the maximum number of concurrent tasks by calling:

  gcloud tasks queues update QUEUE_ID \
          --max-concurrent-dispatches=MAX_CONCURRENT_DISPATCHES

Describe the queue:

  gcloud tasks queues describe QUEUE_ID --location=LOCATION

Replace the following:

  • QUEUE_ID: the queue ID (its short name)
  • LOCATION: the location of the queue

The output should be:

name: projects/PROJECT_ID/locations/LOCATION_ID/queues/QUEUE_ID
rateLimits:
  maxBurstSize: 100
  maxConcurrentDispatches: MAX_CONCURRENT_DISPATCHES
  maxDispatchesPerSecond: 500.0
retryConfig:
  maxAttempts: 100
  maxBackoff: 3600s
  maxDoublings: 16
  minBackoff: 0.100s
state: RUNNING

Defining processing rates using gcloud commands versus using queue.yaml

The Cloud Tasks API approach for defining queue processing rates differs slightly from the approach taken using the uploading of queue.yaml files, even though both methods result in queues using the same underlying mechanism.

In both cases, the queue uses the token bucket algorithm to control the rate of task execution. Each named queue has a bucket that holds its tokens.

Each time your application executes a task, a token is removed from the bucket. The queue continues processing tasks until its bucket runs out of tokens. The system refills the bucket with new tokens continuously based on the max_dispatches_per_second rate that you specify for the queue. If your queue contains tasks to process, and the queue's bucket contains tokens, the system simultaneously processes as many tasks as there are tokens, up to the max_concurrent_dispatches value you have set.

Uneven load can allow the number of tokens in the bucket to grow significantly, which can lead to bursts of processing when a burst of requests then comes in. In this case, your queue may experience an actual dispatch rate that exceeds your max_dispatches_per_second rate, consuming system resources and competing with user-serving requests. In cases where you are using queues to manage dispatch rates based on relatively slow SLAs for downstream services, this can lead to errors like HTTP 429 (Too Many Requests) or 503 (Service Unavailable).

When you use any Cloud Tasks API method, you have two fields to define the queue dispatch rate:

  • max_dispatches_per_second
  • max_concurrent_dispatches

A third field, max_burst_size, is calculated by the system based on the value you set for max_dispatches_per_second.

When you use the queue.yaml method, you can set all three elements:

  • max_concurrent_requests, which is equivalent to max_concurrent_dispatches
  • rate, which is equivalent to max_dispatches_per_second
  • bucket_size, which is equivalent to max_burst_size

In most cases, using the Cloud Tasks API method and letting the system set max_burst_size produces a very efficient rate for managing request bursts. In some cases, however, particularly when the desired rate is relatively slow, either using the queue.yaml method to manually set bucket_size to a small value, or setting your max_concurrent_dispatches to a small value via the Cloud Tasks API can give you more control.

Set retry parameters

If a task does not complete successfully, then Cloud Tasks will retry the task with exponential backoff according to the parameters you have set. You can specify the maximum number of times to retry failed tasks in the queue, set a time limit for retry attempts, and control the interval between attempts.

  gcloud tasks queues update QUEUE_ID \
      --max-attempts=MAX_ATTEMPTS \
      --min-backoff=MIN_INTERVAL \
      --max-backoff=MAX_INTERVAL \
      --max-doublings=MAX_DOUBLINGS \
      --max-retry-duration=MAX_RETRY_DURATION

Replace the following:

  • MAX_ATTEMPTS: the maximum number of attempts for a task, including the first attempt. You can allow unlimited retries by setting this flag to unlimited.
  • MIN_INTERVAL: the minimum amount of time to wait between retry attempts. The value must be a string that ends in "s," such as 5s.
  • MAX_INTERVAL: the maximum amount of time to wait between retry attempts. The value must be a string that ends in "s," such as 5s.
  • MAX_DOUBLINGS: the maximum number of times that the interval between failed task retries will be doubled before the increase becomes constant.
  • MAX_RETRY_DURATION: the maximum amount of time for retrying a failed task measured from when the task was first attempted. The value must be a string that ends in "s," such as 5s.

Verify your queue was configured successfully:

    gcloud tasks queues describe QUEUE_ID --location=LOCATION

Replace the following:

  • QUEUE_ID: the queue ID (its short name)
  • LOCATION: the location of the queue

What's next