Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add read task for gcs #155

Merged
merged 7 commits into from
Jun 24, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
chore: add comment and modify the document
  • Loading branch information
chuang8511 committed Jun 17, 2024
commit dcb581cd2e0e8a6c7a539c6e8efa48da8ac80bef
46 changes: 45 additions & 1 deletion data/googlecloudstorage/v0/README.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ The Google Cloud Storage component is a data component that allows users to uplo
It can carry out the following tasks:

- [Upload](#upload)
- [Read Objects](#read-objects)
- [Create Bucket](#create-bucket)

## Release Stage

Expand All @@ -22,7 +24,6 @@ The component configuration is defined and maintained [here](https://github.com/

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Bucket Name (required) | `bucket-name` | string | Name of the bucket to be used for object storage. |
| JSON Key File contents (required) | `json-key` | string | Contents of the JSON key file with access to the bucket. |

## Supported Tasks
Expand All @@ -34,6 +35,7 @@ Upload data to Google Cloud Storage.
| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_UPLOAD` |
| Bucket Name (required) | `bucket-name` | string | Name of the bucket to be used for object storage. |
| Object Name (required) | `object-name` | string | The name of the object to be created |
| Data (required) | `data` | string | The data to be saved in the object |

Expand All @@ -44,3 +46,45 @@ Upload data to Google Cloud Storage.
| Public Access (optional) | `public-access` | boolean | Whether the object is publicly accessible |
| Public URL (optional) | `public-url` | string | Anyone with this link can access the object on the public Internet |
| Upload Status | `status` | string | Status of the upload operation |

### Read Objects

Read objects from Google Cloud Storage.

| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_READ_OBJECTS` |
| Bucket Name (required) | `bucket-name` | string | Name of the bucket to be used for object storage. |
| Delimiter | `delimiter` | string | The delimiter to use when listing objects |
| Prefix | `prefix` | string | The prefix to use when listing objects |
| Versions | `versions` | boolean | Whether to list all versions of an object |
| Start Offset | `start-offset` | string | The offset to start listing objects from |
| End Offset | `end-offset` | string | The offset to stop listing objects at |
| Include Trailing Delimiter | `include-trailing-delimiter` | boolean | Whether to include the delimiter in the object name |
| Match Glob | `match-glob` | string | Whether to match objects using a glob pattern |
| Include Folders As Prefixes | `include-folders-as-prefixes` | boolean | Whether to include folders as prefixes |

| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Audio Objects (optional) | `audio-objects` | array[object] | The audio objects in the bucket |
| Document Objects (optional) | `document-objects` | array[object] | The document objects in the bucket |
| Image Objects (optional) | `image-objects` | array[object] | The image objects in the bucket |
| Text Objects (optional) | `text-objects` | array[object] | The text objects in the bucket |
| Video Objects (optional) | `video-objects` | array[object] | The video objects in the bucket |

### Create Bucket

Read objects from Google Cloud Storage.

| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CREATE_BUCKET` |
| Bucket Name (required) | `bucket-name` | string | Name of the bucket to be used for object storage. |
| Project ID (required) | `project-id` | string | The project ID to create the bucket in |
| Location | `location` | string | The location to create the bucket in |

| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Result (optional) | `result` | string | The result of the operation |
| Bucket Name (optional) | `name` | string | The name of the bucket |
| Location (optional) | `location` | string | The location of the bucket |
4 changes: 0 additions & 4 deletions data/googlecloudstorage/v0/create_bucket.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package googlecloudstorage

import (
"context"
"fmt"

"cloud.google.com/go/storage"
)
Expand All @@ -28,13 +27,10 @@ func createBucket(input CreateBucketInput, client *storage.Client, ctx context.C
Location: input.Location,
}

fmt.Println("Creating bucket with name: ", input.BucketName)
if err := bkt.Create(ctx, input.ProjectID, &attr); err != nil {
fmt.Println("Error creating bucket: ", err)
return output, err
}


newBktAttrs, err := bkt.Attrs(ctx)
if err != nil {
return output, err
Expand Down
4 changes: 4 additions & 0 deletions data/googlecloudstorage/v0/main_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
package googlecloudstorage

// TODO: chuang8511: add test cases by mocking the GCS client
// It will be done in 2024-06-26
Loading