[gcloud] Add configurable blob parameters (jschneier#970)

* [gcloud] Clean up documentation

* [gcloud] Add configurable blob parameters

docs/backends/gcloud.rst

@@ -38,24 +38,29 @@ Alternatively, you can use the setting `GS_CREDENTIALS` as described below.
 
 Getting Started
 ---------------
-Set the default storage and bucket name in your settings.py file:
+Set the default storage and bucket name in your settings.py file
 
-::
+.. code-block:: python
 
  DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
  GS_BUCKET_NAME = 'YOUR_BUCKET_NAME_GOES_HERE'
 
+
 To allow ``django-admin`` collectstatic to automatically put your static files in your bucket set the following in your settings.py::
 
  STATICFILES_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
 
-Once you're done, default_storage will be Google Cloud Storage::
+Once you're done, default_storage will be Google Cloud Storage
+
+.. code-block:: python
 
  >>> from django.core.files.storage import default_storage
  >>> print(default_storage.__class__)
  <class 'storages.backends.gcloud.GoogleCloudStorage'>
 
-This way, if you define a new FileField, it will use the Google Cloud Storage::
+This way, if you define a new FileField, it will use the Google Cloud Storage
+
+.. code-block:: python
 
  >>> from django.db import models
  >>> class Resume(models.Model):
@@ -69,10 +74,6 @@ This way, if you define a new FileField, it will use the Google Cloud Storage::
 Settings
 --------
 
-To use gcloud set::
-
- DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
-
 ``GS_BUCKET_NAME``
 
 Your Google Storage bucket name, as a string. Required.
@@ -82,6 +83,8 @@ Your Google Storage bucket name, as a string. Required.
 Your Google Cloud project ID. If unset, falls back to the default
 inferred from the environment.
 
+.. _gs-creds:
+
 ``GS_CREDENTIALS`` (optional)
 
 The OAuth 2 credentials to use for the connection. If unset, falls
@@ -96,6 +99,8 @@ back to the default inferred from the environment
  "path/to/credentials.json"
  )
 
+.. _gs-default-acl:
+
 ``GS_DEFAULT_ACL`` (optional, default is None)
 
 ACL used when creating a new blob, from the
@@ -104,7 +109,7 @@ ACL used when creating a new blob, from the
 translated.)
 
 For most cases, the blob will need to be set to the ``publicRead`` ACL in order for the file to be viewed.
-If GS_DEFAULT_ACL is not set, the blob will have the default permissions set by the bucket.
+If ``GS_DEFAULT_ACL`` is not set, the blob will have the default permissions set by the bucket.
 
 ``publicRead`` files will return a public, non-expiring url. All other files return
 a signed (expiring) url.
@@ -125,10 +130,6 @@ If set to ``False`` it forces the url not to be signed. This setting is useful i
 bucket configured with ``Uniform`` access control configured with public read. In that case you should
 force the flag ``GS_QUERYSTRING_AUTH = False`` and ``GS_DEFAULT_ACL = None``
 
-``GS_FILE_CHARSET`` (optional)
-
-Allows overriding the character set used in filenames.
-
 ``GS_FILE_OVERWRITE`` (optional: default is ``True``)
 
 By default files with the same name will overwrite each other. Set this to ``False`` to have extra characters appended.
@@ -147,9 +148,31 @@ must fit in memory. Recommended if you are going to be uploading large files.
 
  This must be a multiple of 256K (1024 * 256)
 
-``GS_CACHE_CONTROL`` (optional: default is ``None``)
+``GS_OBJECT_PARAMETERS`` (optional: default is ``{}``)
+
+Dictionary of key-value pairs mapping from blob property name to value.
+
+Use this to set parameters on all objects. To set these on a per-object
+basis, subclass the backend and override ``GoogleCloudStorage.get_object_parameters``.
 
-Sets Cache-Control HTTP header for the file, more about HTTP caching can be found `here <https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching#cache-control>`_
+The valid property names are ::
+
+ acl
+ cache_control
+ content_disposition
+ content_encoding
+ content_language
+ content_type
+ metadata
+ storage_class
+
+If not set, the ``content_type`` property will be guessed.
+
+If set, ``acl`` overrides :ref:`GS_DEFAULT_ACL <gs-default-acl>`.
+
+.. warning::
+
+ Do not set ``name``. This is set automatically based on the filename.
 
 ``GS_CUSTOM_ENDPOINT`` (optional: default is ``None``)
 
@@ -165,7 +188,7 @@ Defaults to the root of the bucket.
 
 The time that a generated URL is valid before expiration. The default is 1 day.
 Public files will return a url that does not expire. Files will be signed by
-the credentials provided to django-storages (See GS_CREDENTIALS).
+the credentials provided to django-storages (See :ref:`GS Credentials <gs-creds>`).
 
 Note: Default Google Compute Engine (GCE) Service accounts are
 `unable to sign urls <https://googlecloudplatform.github.io/google-cloud-python/latest/storage/blobs.html#google.cloud.storage.blob.Blob.generate_signed_url>`_.
@@ -180,13 +203,17 @@ Usage
 Fields
 ^^^^^^
 
-Once you're done, default_storage will be Google Cloud Storage::
+Once you're done, default_storage will be Google Cloud Storage
+
+.. code-block:: python
 
  >>> from django.core.files.storage import default_storage
  >>> print(default_storage.__class__)
  <class 'storages.backends.gcloud.GoogleCloudStorage'>
 
-This way, if you define a new FileField, it will use the Google Cloud Storage::
+This way, if you define a new FileField, it will use the Google Cloud Storage
+
+.. code-block:: python
 
  >>> from django.db import models
  >>> class Resume(models.Model):
@@ -200,7 +227,9 @@ This way, if you define a new FileField, it will use the Google Cloud Storage::
 Storage
 ^^^^^^^
 
-Standard file access options are available, and work as expected::
+Standard file access options are available, and work as expected
+
+.. code-block:: python
 
  >>> default_storage.exists('storage_test')
  False
@@ -222,7 +251,9 @@ Standard file access options are available, and work as expected::
 Model
 ^^^^^
 
-An object without a file has limited functionality::
+An object without a file has limited functionality
+
+.. code-block:: python
 
  >>> obj1 = Resume()
  >>> obj1.pdf
@@ -232,7 +263,9 @@ An object without a file has limited functionality::
  ...
  ValueError: The 'pdf' attribute has no file associated with it.
 
-Saving a file enables full functionality::
+Saving a file enables full functionality
+
+.. code-block:: python
 
  >>> obj1.pdf.save('django_test.txt', ContentFile('content'))
  >>> obj1.pdf
@@ -242,7 +275,9 @@ Saving a file enables full functionality::
  >>> obj1.pdf.read()
  'content'
 
-Files can be read in a little at a time, if necessary::
+Files can be read in a little at a time, if necessary
+
+.. code-block:: python
 
  >>> obj1.pdf.open()
  >>> obj1.pdf.read(3)
@@ -252,7 +287,9 @@ Files can be read in a little at a time, if necessary::
  >>> '-'.join(obj1.pdf.chunks(chunk_size=2))
  'co-nt-en-t'
 
-Save another file with the same name::
+Save another file with the same name
+
+.. code-block:: python
 
  >>> obj2 = Resume()
  >>> obj2.pdf.save('django_test.txt', ContentFile('more content'))
@@ -261,7 +298,9 @@ Save another file with the same name::
  >>> obj2.pdf.size
  12
 
-Push the objects into the cache to make sure they pickle properly::
+Push the objects into the cache to make sure they pickle properly
+
+.. code-block:: python
 
  >>> cache.set('obj1', obj1)
  >>> cache.set('obj2', obj2)

storages/backends/gcloud.py

@@ -1,4 +1,5 @@
 import mimetypes
+import warnings
 from datetime import timedelta
 from tempfile import SpooledTemporaryFile
 
@@ -23,6 +24,9 @@
  "See https://github.com/GoogleCloudPlatform/gcloud-python")
 
 
+CONTENT_TYPE = 'content_type'
+
+
 class GoogleCloudFile(File):
  def __init__(self, name, mode, storage):
  self.name = name
@@ -106,6 +110,7 @@ def get_default_settings(self):
  "expiration": setting('GS_EXPIRATION', timedelta(seconds=86400)),
  "file_overwrite": setting('GS_FILE_OVERWRITE', True),
  "cache_control": setting('GS_CACHE_CONTROL'),
+ "object_parameters": setting('GS_OBJECT_PARAMETERS', {}),
  # The max amount of memory a returned file can take up before being
  # rolled over into a temporary file on disk. Default is 0: Do not
  # roll over.
@@ -154,12 +159,34 @@ def _save(self, name, content):
 
  content.name = cleaned_name
  file = GoogleCloudFile(name, 'rw', self)
- file.blob.cache_control = self.cache_control
- file.blob.upload_from_file(
- content, rewind=True, size=content.size,
- content_type=file.mime_type, predefined_acl=self.default_acl)
+
+ upload_params = {}
+ blob_params = self.get_object_parameters(name, content)
+ if 'cache_control' not in blob_params and self.cache_control:
+ warnings.warn(
+ 'The GS_CACHE_CONTROL setting is deprecated. Use GS_OBJECT_PARAMETERS to set any '
+ 'writable blob property or override GoogleCloudStorage.get_object_parameters to '
+ 'vary the parameters per object.', DeprecationWarning
+ )
+ blob_params['cache_control'] = self.cache_control
+
+ upload_params['predefined_acl'] = blob_params.pop('acl', self.default_acl)
+ if CONTENT_TYPE not in blob_params:
+ upload_params[CONTENT_TYPE] = file.mime_type
+
+ for prop, val in blob_params.items():
+ setattr(file.blob, prop, val)
+
+ file.blob.upload_from_file(content, rewind=True, size=content.size, **upload_params)
  return cleaned_name
 
+ def get_object_parameters(self, name, content):
+ """Override this to return a dictionary of overwritable blob-property to value.
+
+ Returns GS_OBJECT_PARAMETRS by default. See the docs for all possible options.
+ """
+ return self.object_parameters.copy()
+
  def delete(self, name):
  name = self._normalize_name(clean_name(name))
  try: