Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rfc(decision): Batch multiple files together into single large file to improve network throughput #98

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open

rfc(decision): Batch multiple files together into single large file to improve network throughput #98

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
912301d
rfc(decision): Store Multiple Replay Segments in a Single Blob
cmanallen May 24, 2023
7dfe76c
Initial commit
cmanallen May 24, 2023
f82f9e6
Add semicolons
cmanallen May 24, 2023
721c28c
Fix flowchart
cmanallen May 24, 2023
4ff4376
Use correct byte start
cmanallen May 24, 2023
4ec773a
Add extension
cmanallen May 24, 2023
86c1819
Remove ancillary goal motivation
cmanallen May 24, 2023
9f9d123
Add more detail
cmanallen May 25, 2023
8b25e3b
Use clearer language
cmanallen May 25, 2023
d34401e
Add technical details section
cmanallen May 26, 2023
8af3d50
Merge branch 'rfc/store-multiple-replay-segments-in-a-single-blob' of…
cmanallen May 26, 2023
812a110
Re-write to include notes about encryption of files
cmanallen Jun 8, 2023
f5fe8e8
Update questions section
cmanallen Jun 8, 2023
78a0d6e
Add created-at column
cmanallen Jun 8, 2023
82f222d
Add high-level overview
cmanallen Jun 8, 2023
a7ad1d6
Improve formatting
cmanallen Jun 9, 2023
d63a30a
Clean up managing effects section slightly
cmanallen Jun 9, 2023
354431d
Add more to supporting data section and add an FAQ
cmanallen Jul 18, 2023
15430b4
Re-write RFC to be product-neutral
cmanallen Aug 29, 2023
71f1715
Add conclusion
cmanallen Aug 29, 2023
3759346
To to with
cmanallen Aug 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add high-level overview
  • Loading branch information
@cmanallen
cmanallen committed Jun 8, 2023
commit 82f222da818902458342a709868acaa74153b228
22 changes: 21 additions & 1 deletion text/0098-store-multiple-replay-segments-in-a-single-blob.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Recording data is sent in segments. Each segment is written to its own file. Wri
# Motivation

1. Minimize costs.
2. Improve throughput.
2. Increase write throughput.
3. Enable new features in a cost-effective manner.

# Background
Expand All @@ -23,6 +23,26 @@ Google Cloud Storage lists the costs for writing and storing data as two separat

In practical terms, this means 75% of our spend is allocated to writing new files.
cmanallen marked this conversation as resolved.
Show resolved Hide resolved
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While the cost is mostly due to write, at scale is this cost problematic? Like is it a blocker for reaching higher scale or are we simply looking for a more efficient option ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not believe cost will prevent us from reaching greater scale. Write cost scale linearly. Pricing does not quite scale linearly but if you're happy with how much GCS costs at low levels of demand you will be happy at peak demand.


# High Level Overview

1. We will store multiple "parts" per file.
- A "part" is a distinct blob of binary data.
- It exists as a subset of bytes within a larger set of bytes (referred to as a "file").
- A "part" could refer to a replay segment or to a sourcemap or anything that requires storage in a blob storage service.
2. Each "part" within a file will be encrypted.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since encryption is used here entirely to make deletion of individual parts quicker, could you please expand on:

  • whether we are confident that this is a use case worth optimizing for (deleting individual part quickly)
  • how would the process look like if we just accepted to rewrite files entirely in order to remove the part.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whether we are confident that this is a use case worth optimizing for (deleting individual part quickly)

It is necessary to support GDPR, project, and point deletes. I consider it a high priority. The alternative is rotating the file with the offending parts removed.

how would the process look like if we just accepted to rewrite files entirely in order to remove the part.

The operations is no particular order:

  1. Download the file
  2. Get the active byte ranges from the database.
  3. Remove all byte ranges from the file not found in the set of returned byte ranges.
  4. Delete all references to the file in the database.
  5. Upload the new file.
  6. Insert new offset rows.
  7. Delete the old file.

Repeat for every delete operation. Deletes must be single-threaded per file to prevent concurrent access. You can use kafka and partition on filename.

- Encryption provides instantaneous deletes (by deleting the row containing the encryption key) and removes the need to remove the byte sub-sequences from a blob.
- We will use envelope encryption to protect the contents of every file.
- https://cloud.google.com/kms/docs/envelope-encryption
- Related, contiguous byte ranges will be encrypted independently of the rest of the file.
- We will use KMS to manage our key-encryption-keys.
Copy link

@fpacifici fpacifici Jul 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KMS ?
Are you talking about GCP KMS ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Google Key Management Service. Subject to change.

- Data-encryption-keys will be generated locally and will be unique.
3. Parts will be tracked in a metadata table on an AlloyDB instance(s).
- A full table schema is provided in the **Proposal** section.
- AlloyDB was chosen because its a managed database with strong point-query performace.
- The metadata table will contain the key used to decrypt the byte range.
4. On read, parts will be fetched without fetching the full file.
- More details are provided in the **Technical Details** section.

# Proposal

First, a new table called "file_part_byte_range" with the following structure is created:
Expand Down