Add PUT uploads to object storage client #25
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
br3ndonland
added a commit
that referenced
this pull request
Jan 29, 2024
Dotenv files are commonly kept in cloud object storage. fastenv provides an object storage client for downloading and uploading dotenv files. https://fastenv.bws.bio/cloud-object-storage The fastenv object storage client currently supports AWS S3 and Backblaze B2. There are many other object storage platforms with "S3-compatible" APIs, including Cloudflare R2. This commit will implement support for Cloudflare R2. https://developers.cloudflare.com/r2/ - Handle Cloudflare R2 bucket hosts - The fastenv object storage client works with "virtual-hosted-style" URLs, as this is the preferred format for AWS S3 - Cloudflare R2 "virtual-hosted-style" URLs, like `https://<BUCKET>.<ACCOUNT_ID>.r2.cloudflarestorage.com`, were implemented on 2022-05-16. - Handle Cloudflare R2 bucket region `auto` https://developers.cloudflare.com/r2/api/s3/api/ - Add Cloudflare R2 credentials to tests - Skip tests that do uploads with `POST` (Cloudflare R2 only supports uploads with `PUT` https://developers.cloudflare.com/r2/api/s3/presigned-urls/ - Update object storage docs with info on Cloudflare R2 #8 #25 [Cloudflare R2](https://developers.cloudflare.com/r2/)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Dotenv files are commonly kept in cloud object storage. fastenv provides an object storage client for downloading and uploading dotenv files.
S3-compatible object storage allows uploads with either
POSTorPUT. This PR will implement uploads withPUT.Changes
This PR will add a
methodargument tofastenv.ObjectStorageClient.upload()that accepts eitherPOSTorPUT.POSTwas previously the default, butPUTwill now be the default becausePUTuploads are more widely-supported and standardized. Backblaze B2 does not currently support single-part uploads withPOSTto their S3 API (the B2 native API must be used instead), and Cloudflare R2 does not support uploads withPOSTat all.Files will be opened in binary mode and attached with the
contentargument (httpx_client.put(content=content)) as suggested in the HTTPX docs.Unlike downloads with
GET, presignedPUTURL query parameters do not necessarily contain all the required information. Additional information may need to be supplied in request headers. In addition to supplying header keys and values with HTTP requests, header keys should be signed into the URL in theX-Amz-SignedHeadersquery string parameter (alphabetically-sorted, semicolon-separated, lowercased).These request headers can specify:
X-Amz-Server-Side-Encryption. Note that, although similar headers likeX-Amz-Algorithmare included as query string parameters in presigned URLs,X-Amz-Server-Side-Encryptionis not. IfX-Amz-Server-Side-Encryptionis included in query string parameters, it may be silently ignored by the object storage platform. AWS S3 now automatically encrypts all objects and Cloudflare R2 does also, but Backblaze B2 will only automatically encrypt objects if the bucket has default encryption enabled.Content-MD5header, defined by RFC 1864, can supply a base64-encoded MD5 checksum. After the upload is completed, the object storage platform server will calculate a checksum for the object in the same manner. If the client and server checksums are the same, this means that all expected information was successfully sent to the server. If the checksums are different, this may mean that object information was lost in transit, and an error will be reported. Note that, although Backblaze B2 accepts and processes theContent-MD5header, it will report a SHA1 checksum to align with uploads to the B2-native API.Content-Disposition,Content-Length, andContent-Typecan be supplied in request headers.Related